Adjustable Parallax Distance, Wide Field of View, Stereoscopic Imaging System

ABSTRACT

An imaging system and methods for using an imaging system where the operator is able to variably adjust the parallax distance for enhanced stereo performance are disclosed. In addition, by coordinating the parallax distance with the optical settings of the camera, artificial 3D experiences can be created that give a user the perception of observing a scene from a distance different than that actually employed. The imaging system may also include a plurality of stereo camera supersets, wherein a first one or more stereo camera supersets are positioned at a different height relative to a first stereo camera superset. Novel specific uses of the camera system, such as in capturing events of interest are described. Useful techniques for extracting or encoding wide field of view images from memory are also disclosed.

The present U.S. utility patent application is a continuationapplication of U.S. Ser. No. 14/307,077 filed on Jun. 17, 2014 entitled“Adjustable Parallax Distance, Wide Field of View, Stereoscopic ImagingSystem, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD Background Art

U.S. Pat. Nos. 7,982,777 and 8,004,558, for inventions of the presentinventor with others are hereby incorporated herein by reference intheir entirety. They teach a biologically inspired arrangement ofcameras (and/or camera pairs) that mimics the location of the eyes of abinocular animal (such as a human) as the head is turned side-to-side orup-and-down.

A differentiating characteristic of the above technology is that, due tothe arrangement of cameras, it creates potential obscuration betweenneighboring cameras. This is done to realize the most simple and compactconfiguration (as a function of parallax distance) possible. And Prechtlet. al. describe, at length, techniques by which this obscuration can beaddressed and managed.

A significant parameter affecting the performance of a stereo camerasystem is the parallax distance employed. Some practitioners haveaddressed the possibility of varying the parallax distance in stereo,such as in U.S. Pat. No. 7,224,382 and US Pat. Apps. Nos. 2013/0044181and 2013/081576.

The era of video-on-demand (VOD) is well established. Consumers are ableto purchase a wide variety of videos to consume, as desired, as part oftheir multimedia collections. Some prior art describes the capture of ascene from multiple cameras in different ways. This is taught in U.S.Pat. No. 8,326,113 and US Patent Application No. 2008/0275881.

There is also a large body of art that addresses the problem of managingelectronic media. This is taught in U.S. Pat. Nos. 7,243,364, 8,527,549,8,544,047, 8,548,735, and 8,676,034.

There is also art related to measuring the size of the audience watchingan event, such as that taught in U.S. Pat. No. 8,249,992.

SUMMARY OF THE EMBODIMENTS

Embodiments of the present invention provide an improvement to thestereoscopic imaging system taught by Prechtl et. al. in U.S. Pat. No.7,982,777 and U.S. Pat. No. 8,004,558. Various embodiments provide animaging system where the operator is able to adjust the parallaxdistance for enhanced stereo performance. In addition, by coordinatingthe parallax distance with the optical settings of the camera,artificial 3D experiences can be created that give a user the perceptionof observing a scene from a distance different than that actuallyemployed. Variable parallax distance embodiments are described that useadjustable mechanisms, static camera configurations or a combination ofthe two.

In addition, embodiments of the wide field-of-view system are describedin novel applications such as for capturing live action, sports &entertainment, and for general security. Algorithms are also presenteddescribing methods by which images may be encoded to compensate for themotion of the capturing platform.

In a first embodiment of the invention, a structure is described thatincorporates a mechanism that can be used to adjust the parallaxdistance of a stereoscopic camera system.

In a second embodiment of the invention, a structure is described thatarranges multiple supersets of cameras wherein each superset captureswide field of view stereoscopic images of a scene from a differentparallax distance. In this second embodiment, common arm structures areused to mount similarly oriented digital cameras from differentsupersets.

In a third embodiment of the invention, a structure is described thatarranges multiple supersets of cameras wherein each superset captureswide field of view stereoscopic images of a scene from a differentparallax distance. In this third embodiment, separate arm structures areused for each superset.

In a fourth embodiment, an application of wide field-of-view imaging isused to capture events of interest. A novel process by which scenes canbe captured and saved by multiple end-users, simultaneously, isdescribed.

In a fifth embodiment, a method of encoding orientation data along withvideo data from a wide field of view camera system that is moving isdescribed. The orientation data can be used to more smoothly access andview the video data.

More particularly, in one embodiment, the invention provides a widefield-of-view stereoscopic camera system. In this embodiment, the systemincludes:

a support structure having M arms, where M is an integer equal to orgreater than 2, each of the arms being spaced around a constructivecircle and having a mount point, coincident with the circle, defining aradius of the circle, wherein the radius is half of a parallax distance,and the circle defines a horizontal plane and the circle defines aconstructive right cylinder that contains that circle and that has anaxis perpendicular to the horizontal plane and wherein each of the armshas a length adjustment to vary the distance between the center of thecircle and the mount point;

N digital cameras, wherein N is at least equal to M, each digital camerahaving a lens with a focal point and an optical axis and producing animage encoded in a digital output and having a horizontal field of viewand a vertical field of view;

the digital cameras mounted to the support structure so that each armsupports at least one digital camera so that (i) the focal point thereofis proximate to such arm's mount point and (ii) the optical axis thereofis generally tangent to the cylinder; and

an image processor, coupled to the digital output of each of the Ndigital cameras, to form a stereo view pair, wherein the processor formsa left view from the digital outputs of a first set of the N cameras andforms a right view from the digital outputs of a second set of the Ncameras, wherein the first and second sets are disjoint.

In a related embodiment, at least one digital camera is mounted so thatits horizontal field of view is subject to a partial obscuration by anobject associated with an adjacent arm, and wherein the image processorcompensates for the obscuration at least in part by a view availablefrom one of the digital cameras mounted on an adjacent arm.

Optionally, the digital camera subject to the partial obscuration ismounted with an outward angle such that the partial obscuration isminimized. Also optionally, the left view and the right view both coverthe 360 degree horizontal field of view. As a further option 1 or moreof the N digital cameras are mounted such that the optical axis is notparallel with the horizontal plane.

In a further related embodiment, 2 or more digital cameras are mountedproximal to at least one mount point such that their vertical fields ofview partially overlap to capture an extended elevation field of view.

In another related embodiment, each of the arms is pivotally coupled toa mounting pole that passes through the center of the circle and isdisposed perpendicularly with respect to the horizontal plane, each ofthe arms having a link pivotally attached to a circumferential supportthat is slidably coupled to the mounting pole, so that motion of thecircumferential support along the mounting pole adjusts the radius ofthe circle and therefore the parallax distance.

In yet another related embodiment, each of the arms is coupled to asliding arm mechanism that allows the arms to move in substantiallyradial direction and that includes a rack and pinion, so that rotationalmotion of the pinion around its central axis adjusts the radius of thecircle and therefore the parallax distance.

Optionally, the system further includes a motorized drive coupled to atleast one of the arms, and a controller coupled to the motorized drive,so as to cause adjustment in length of at least one of the arms, and thecontroller is coupled to one of the cameras on the at least one of thearms to receive from the camera at least one parameter of the camera andto cause an adjustment in length of at least one of the arms responsiveto a change in the at least one parameter.

In another related embodiment, the processing algorithm makes use ofprojective transformations to form the left view and the right view.Optionally, the processing algorithm averages out the image quality ofthe aggregate set of pixels from the images of all N cameras to create asmooth transition in color, contrast, or white levels in the left viewand the right view.

In a further related embodiment, the stereo pair of views is formattedto be compatible with a 3D display technology.

In another related embodiment, the image processor further operates onthe left view and the right view to estimate the distance of items fromthe camera system using stereo triangulation.

Optionally, the system is used in the application of teleroboticplatform control, surveillance, object tracking, facial recognition,access control, traffic control, collision avoidance, sports orentertainment event viewing, mapping or surveying, telepresence, realestate promotion, restaurant promotion, hotel promotion, remote tourism,thrill rides, gaming, or training.

In another embodiment, the invention provides a wide field-of-viewstereoscopic camera system. This embodiment includes:

a support structure having L sets of arms, where L is an integer greaterthan 1, and each set has at least 2 arms, each of the arms in aparticular one of the sets being spaced around a distinct one of Lconstructive circles, and having a mount point, coincident with thedistinct one circle, defining a radius of the circle, wherein the radiusis half of a parallax distance, and the distinct one circle defines aplane that is parallel to a horizontal plane and a constructive rightcylinder containing the distinct one circle and having an axisperpendicular to the horizontal plane;

N digital cameras, wherein N is at least equal to the number of arms inthe camera system, each digital camera having a lens with a focal pointand an optical axis and producing an image encoded in a digital outputand having a horizontal field of view and a vertical field of view,

the digital cameras mounted to the support structure so that each armsupports at least one digital camera so that (i) the focal point thereofis proximate to such arm's mount point and (ii) the optical axis thereofis generally tangent to the cylinder corresponding to that mount point;and

an image processor, coupled to the digital output of each of the Ndigital cameras, that forms L independent stereo view pairs, wherein,for each stereo view pair, the processor forms a left view from thedigital outputs of a first set of the N cameras and forms a right viewfrom the digital outputs of a second set of the N cameras, wherein thefirst and second sets are disjoint.

Optionally, at least one digital camera is mounted so that itshorizontal field of view is subject to a partial obscuration by anobject associated with an adjacent arm, and wherein the image processorcompensates for the obscuration at least in part by a view availablefrom one of the digital cameras mounted on an adjacent arm. As a furtheroption, the digital camera subject to the partial obscuration is mountedwith an outward angle such that the partial obscuration is minimized.

In another related embodiment, the left view and the right view of atleast one set of L cameras both cover the 360 degree horizontal field ofview.

In another related embodiment, 1 or more of the N digital cameras aremounted such that the optical axis is not parallel with the horizontalplane. Optionally, 2 or more digital cameras are mounted proximal to atleast one mount point such that their vertical fields of view partiallyoverlap to capture an extended elevation field of view.

In another related embodiment, the digital cameras mounted proximal tomount points corresponding to one constructive cylinder are furthermounted at an elevation with respect to the horizontal plane such thattheir fields of view are not obscured by any digital camera mountedproximal to a mount point that corresponds to a different constructivecylinder.

Also optionally, each of the arms in at least one of the L sets of armshas a length adjustment to vary the distance between the center of thecircle and the mount point.

Also optionally, the image processor further operates on the left viewand the right view, of one or more of the L sets of digital cameras, toestimate the distance of items from the camera system using stereotriangulation.

Alternatively or in addition, the processor, responsive to a signalreceived by the processor, provides a selected one of the stereo viewpairs as an output.

Optionally, each of the L stereo pair of views is formatted to becompatible with a 3D display technology.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of embodiments will be more readily understood byreference to the following detailed description, taken with reference tothe accompanying drawings, in which:

FIG. 1 is an isometric view of the camera system design according to theprior art of Prechtl et. al. with a parallax distance that matches theinterpupillary distance of the human head.

FIG. 2 is a top view of the prior art camera system illustrated in FIG.1.

FIG. 3 is an illustration of the underlying geometry of a general camerasystem design with an even number of evenly spaced mount points inaccordance with an embodiment the present invention.

FIG. 4 is an illustration of the underlying geometry of a general camerasystem design with an odd number of evenly spaced mount points inaccordance with an embodiment the present invention.

FIG. 5 is a top view of a variable parallax distance, 6-arm camerasystem design, at a minimum parallax distance configuration inaccordance with an embodiment the present invention.

FIG. 6 is an isometric view of a variable parallax distance, 6-armcamera system design, at a minimum parallax distance configuration inaccordance with an embodiment the present invention.

FIG. 7 is a isometric section view of the variable parallax distance,6-arm camera system design, at a minimum parallax distanceconfiguration, highlighting the rack-and-pinion transmission inaccordance with an embodiment the present invention.

FIG. 8 is an isometric view of a variable parallax distance, 6-armcamera system design, at an intermediate parallax distance configurationin accordance with an embodiment the present invention.

FIG. 9 is an isometric view of a variable parallax distance, 6-armcamera system design, at a maximum parallax distance configuration inaccordance with an embodiment the present invention.

FIG. 10 is an isometric view of a variable parallax distance, 6-armcamera system design, at a minimum parallax distance configuration withviewing frustums for each of the right eye camera daughter boards andshowing a central electronics component located above the imagingstructure in accordance with an embodiment the present invention.

FIG. 11 is a view of an extended elevation camera bracket structure withthree right eye camera daughter boards and three left eye cameradaughter boards in accordance with an embodiment the present invention.

FIG. 12 is a zoomed out image of the extended elevation camera armstructure of FIG. 11 that includes wire frame viewing frustums for thecamera daughter boards highlighting the extended vertical field of viewcoverage in accordance with an embodiment the present invention.

FIG. 13 is an isometric view of a variable parallax distance, 6-arm,extended elevation camera system design, at an intermediate parallaxdistance configuration. Wire frame frustums for the three right-eyecamera daughter boards at the end of two of the arms are illustrated tohighlight how the frustums cooperate to provide extended elevationperformance in accordance with an embodiment the present invention.

FIG. 14 is a top view of a variable parallax distance, 8-arm camerasystem design, at a minimum parallax distance configuration inaccordance with an embodiment the present invention.

FIG. 15 is an isometric view of a selectable parallax distance, 6-armcamera system design in accordance with an embodiment the presentinvention.

FIG. 16 is a top view of the selectable parallax distance, 6-arm camerasystem design of FIG. 15.

FIG. 17 is an isometric section view of a selectable parallax distance,6-arm camera system design in accordance with an embodiment the presentinvention. Wire frame frustums are drawn to highlight how the arms arecontoured to minimize obscuration between camera supersets.

FIG. 18 is an isometric view of a selectable parallax distance, 18-armextended elevation camera system design composed of multiple levels ofcamera supersets with different parallax distance values in accordancewith an embodiment the present invention.

FIG. 19 is an isometric view of the camera system of FIG. 18 with solidviewing frustums drawn for six minimum parallax distance camera daughterboards to highlight the design placement to minimize obscuration betweencamera supersets.

FIG. 20 is an isometric view of a typical event where a spectator mightwant to capture video clips from a selectable parallax distance widefield-of-view video network in accordance with an embodiment the presentinvention.

FIG. 21 is an illustration of a graphical user interface that could beused to engage with a selectable parallax distance wide field-of-viewvideo network for customized video clip purchase in accordance with anembodiment the present invention.

FIG. 22 is an illustration of a dynamic event where an extendedelevation, wide field-of-view video system could be used to capture anevent, but where the camera orientation is dynamically changing inaccordance with an embodiment the present invention.

FIG. 23 is an illustration of the encoding-to and read-back from memoryof a cropped region of interest from a camera feed where the capturecamera orientation is dynamically changing in accordance with anembodiment the present invention.

FIG. 24 a is an illustration of the underlying geometry of a generalcamera system design with an odd number of unevenly spaced mount points.The right eye viewing frustums are included to support the discussion inaccordance with an embodiment the present invention.

FIG. 24 b is an illustration of the same camera system design of FIG. 24a highlighting the left eye viewing frustums to show the complementaryarrangement.

FIG. 25 illustrates a 6-arm, rack and pinion, variable parallax designwith attached drive motor and control electronics in accordance with anembodiment the present invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS Definitions

As used in this description and the accompanying claims, the followingterms shall have the meanings indicated, unless the context otherwiserequires:

A “digital camera” is a device composed of an image sensor and a lens.The image sensor is further composed of a pixel array, in a plane, thatoperates to convert optical energy from the electromagnetic spectruminto a digital electrical signal. Visible spectrum (390-700 nm) imagesensors are the most familiar. However, embodiments of the inventiondescribed herein can employ image sensors that operate over a wide rangeof the electromagnetic spectrum, including visible, infrared andultraviolet wavelengths. This definition is meant to also capture theso-called “light field” class of image sensors that capture an overalllight field and allow users to refocus images in a post processingfashion (see www.lytro.com for an example). The lens is an opticalelement transparent to the desired region of the electromagneticspectrum. The lens can be a separated element or an array of microlenses distributed above the active pixel array of the image sensor orsome variation to operate on more exotic types of image sensors (such aslight field sensors); in the case of an array of micro lenses, theoptical axis (defined below) will be understood to pass through thecentroid of the focal points of the micro lenses in the array.

A “focal point” is a constructive location in space through whichincoming electromagnetic waves converge before hitting the image sensor.In cases where the digital camera may use a lens with multiple focalpoints (e.g. in cases using an array of micro lenses), for the purposesof this document, the “focal point” is understood to mean a singlereference point within the digital camera that is related to the generalfocus of light rays onto the image sensor.

A digital camera may also be referred to as a “camera daughter board” toacknowledge that it may incorporate additional components that performprocessing tasks, ancillary to the capture and digitizing of theincoming spectrum of light, prior to transmitting the image data to acentral processing hub.

A “miniature digital camera” is a digital camera having a lens mounteddirectly to a digital image sensor or to a digital image sensor circuitboard.

The “frame rate” of an image sensor is the number of times per secondthe active pixels in an image sensor are sampled. For the purposes ofthis document, frame rates can vary from as slow as a single activationat a time (a “snapshot”) up through high speeds, exceeding 100 framesamples per second. Throughout the document, the term ‘video’ is usedfor convenience, suggesting a sequential series of image frames.However, it is understood that this term will include situations wherethe ‘video’ is just a single snapshot. It is further understood that theterm ‘image stream’ will be used synonymously to the word ‘video’.

The “optical axis” of a camera is defined as a line perpendicular to aplane defined by the pixel array of the image sensor, from a point atthe centroid of that array of pixels, passing through a focal point ofthe camera lens, and passing through a point within the imaged scene.

A “horizontal field of view” is an angular measurement of the horizontalarea that is visible to an image sensor. The angular measurement is madein a plane defined by the optical axis and a horizontal line of pixelsin the active pixel array of the corresponding image sensor.

A “vertical field of view’ is an angular measurement of the verticalarea that is visible to an image sensor. The angular measurement is madein a plane defined by the optical axis and a vertical line of pixels inthe active pixel array of the corresponding image sensor. The verticalline of pixels is perpendicular to the horizontal line of pixels used todefine the horizontal field of view.

A “viewing frustum” is a three dimensional rectangular pyramidal regioncharacterized by a horizontal field of view and a vertical field of viewthat are not necessarily equal. The vertex of this pyramidal region isdefined so that respective sides of the frustum correspond to the areavisible by an image sensor. The vertex typically coincides with thefocal point of the digital camera lens.

A “panoramic” view is an image characterized by a horizontal field ofview and vertical field of view as follows: The horizontal field of viewis larger than the horizontal field of view of any one camera in thesystem and can be as large as the full 360 degree horizontal region. Thevertical field of view is less than 90 degrees.

An “omni-directional” view is an image characterized by a horizontalfield of view and vertical field of view as follows: The horizontalfield of view is larger than the horizontal field of view of any onecamera in the system and can be as large as the full 360 degreehorizontal region. The vertical field of view is greater than or equalto 90 degrees and can be as large as the full 180 degree verticalelevation. In the case where the horizontal field of view is 360 degreesand the vertical field of view is 180 degrees, this would define thefull 4π-steradian spherical area surrounding the camera.

A “wide field-of-view” is a panoramic or omni-directional view. In thepreferred embodiment, the stitched output field-of-view of the camerawould be a single cohesive viewing frustum. However, the inventionallows for embodiments where the wide field of view might incorporatemultiple separate large fields-of-view that together will be consideredthe camera's ‘wide’ field-of-view.

A “monoscopic image” is a traditional, 2D representation of a scene. Ittypically signifies the capture of a scene from a single point ofreference.

A “stereoscopic camera system” is composed of a set of cameras such thatthe output of one subset of these cameras can be used to form a left eyeview of an imaged scene simultaneously with the output of a disjointsubset of these cameras being used to form a right eye view ofsubstantially overlapping fields-of-view.

A “stereoscopic image” describes the representation of a scene that wascaptured with a stereoscopic camera system.

A “stereo view pair” is a description of a stereoscopic image and isspecifically composed of a left-eye view of a scene and a right-eye viewof a scene.

The “parallax distance” is defined with respect to a “central point” ofa stereoscopic camera system. One half of the parallax distance definesthe lateral offset from this central point of the focal point of,preferably all but, at least one of the digital cameras in the left-eyeset. In a similar fashion, one half of the parallax distance defines thelateral offset from this central point of the focal point of, preferablyall but, at least one of the digital cameras in the right eye set.

The “parallax circle” is centered on the central point of a stereoscopiccamera system. Its diameter is equal to the parallax distance.

The “parallax cylinder” is a right circular cylinder whose axis passesthrough the central point of a stereoscopic camera system and that isperpendicular to the plane defined by the parallax circle. Its diameteris equal to the parallax distance.

The “interpupillary distance”, abbreviated as ‘IPD’, is the distancebetween the pupils of a subject's two eyes.

The “arms” of the camera system represent structural members thatprovide support for the digital cameras at a specified mount point.While the arms in the illustrated embodiments represent relativelysimple radial elements, it is understood that the ‘arms’ of the camerasystem will include any structural member that provides adequate supportfor the digital cameras while allowing the functional performance of thespecified embodiment.

The optical axis of a digital camera is “generally tangent” to aconstructive right cylinder when the optical axis is oriented so that itis within 30 degrees of a tangent to that cylinder and so that thetangent and the optical axis are co-planar.

The term “Pancam” is a general identifier used for convenience inreferring to the wide field-of-view, stereoscopic camera systems in thisdocument.

Introductory Camera Design

Embodiments of the invention described in this document are focused onthe capture of a wide field-of-view, stereoscopic image with either avariable parallax distance or from the perspective of multiple parallaxdistances, simultaneously. The designs are enabled through a cameraconfiguration previously taught by Prechtl et. al. in U.S. Pat. No.7,982,777 and U.S. Pat. No. 8,004,558. In those descriptions, theparallax distance is treated as a static design parameter. To understandembodiments of this new invention, it is illustrative to brieflyconsider this particular prior art.

FIG. 1 is an isometric view of the camera system design according to theprior art of Prechtl et. al. with a parallax distance that matches theinterpupillary distance of the human head. The camera systemincorporates 6 left eye camera daughter boards 101, 102, 103, 104, 105,and 106, and 6 right eye camera daughter boards 107, 108, 109, 110, 111,and 112 mounted to a support structure 113. Each camera daughter boardcontains a digital camera composed of an image sensor and lens andoptional ancillary processing electronics. The cameras are arranged instereo camera pairs in the figure but such a pairing is not arequirement for operation since the left and right eye camera feeds areprocessed independently.

The central processing electronics for the camera system are not shownin FIG. 1. These components would be mounted just above or just belowthe imaging head 100. In this embodiment, each daughter boardcommunicates its image data to the central processing electronicsthrough a flat flexible cable (FFC). For clarity, only two of the 12 FFCcables are shown in FIG. 1; cable 115, that interfaces with cameradaughter board 103 and cable 116, that interfaces with camera daughterboard 112. A cover 114 is preferably designed to fit on top of thesupport structure with access slots for the FFC cables and vent holes topromote cooling.

FIG. 2 is a top view of the prior art camera system illustrated inFIG. 1. This figure includes wire-frame viewing frustrums illustratedfor the six right eye cameras. The figure shows a zoomed out view in thebottom left and a zoomed in, detail view in the upper right. Viewingfrustums 207, 208, 209, 210, 211, and 212 correspond to camera daughterboards 107, 108, 109, 110, 111, and 112, respectively.

This figure illustrates how each digital camera is mounted such that thefocal point of its lens is coincident with a parallax circle 220, with adiameter 221, p. This particular embodiment was designed to have aparallax distance 221, p, of 63.5 mm, matching the averageinterpupillary distance of the human head. By using a 3 mega-pixel imagesensor in each digital camera, such as OmniVision P/N OV03640-VL9A (soldas part of a Global Digital Star Limited camera (P/N G-Q3B03)), thisdesign can deliver an overall resolution better than 2 arcminutes perpixel, which is the approximate visual acuity of a human. When the imagestream from this camera system is displayed to a human end-user, becauseof the parallax distance and resolution, it will deliver a highlyimmersive experience.

FIG. 2 also highlights the design decision to angle the cameras outwardby a small amount. For example, camera daughter board 111 is orientedsuch that the left side of viewing frustum 211 is approximately tangentto the structure 113 near the outer corner of camera daughter board 101.All cameras daughter boards are oriented in a similar manner. In thepreferred embodiment, the cameras would be oriented so that theiroptical axes are tangent to the parallax circle 220. But this wouldresult in more obscuration than desired in this particularconfiguration.

Prechtl et. al. teach multiple ways to deal with this obscuration. Oneuseful technique is to use the image feeds from a forward camera to fillin the obscured region of the rear camera. So, for example, in thisdesign, a portion of the video feed from camera 110 would be used toreplace the obscured region of camera 111. Identical techniques would beused for all of the cameras in the system. Another technique is tosimply angle the cameras outward. This latter technique is adopted inthe design illustrated in FIG. 1 and FIG. 2.

General Design Considerations

Most of the following discussion assumes a camera design composed ofmultiple equivalent digital cameras. This is done for convenience andshould not be considered a limitation on the design or the embodimentsof the proposed invention. For example, it is possible to build upcamera designs composed of digital cameras with differing levels ofcapabilities and performance. The design considerations of such a systemare addressed at the end of this sub-section.

A Pancam design can be built around two fundamental performancespecifications: the parallax distance, p, and the resolution of imagecapture, r. The resolution, defined in units of arc-minutes per pixel,represents the angular region subtended by each pixel in an image. As areference, the visual acuity of a human is approximately 2arcminutes-per-pixel.

In-Plane Considerations

To satisfy a resolution requirement, the individual digital cameras mustsatisfy a specific relationship between number of sensor pixels and lensfield-of-view. For example, if the image sensor of the digital camerahas n_(x) horizontal pixels in its array, then it must integrate a lenswith a horizontal field-of-view, γ_(x), in degrees, that satisfies

$\gamma_{x} \leq \frac{r\left( n_{x} \right)}{60}$

A similar relation exists relating the number of vertical pixels, n_(y),to the vertical field of view, γ_(y).

The overall camera system will be designed to provide a total horizontalfield of view, Θ, and a total vertical field of view, Φ. Θ is typicallybetween γ_(x) and 360°. Φ is typically between γ_(y) and 180°.

The number of digital cameras (per eye), distributed circumferentially,to satisfy the horizontal field of view requirement at the desiredresolution is¹ ¹ Note that an N_(θ) value less than this could be used,but it would result in incomplete circumferential coverage of the area.A similar point holds for N_(φ), defined below.

$N_{\theta} \geq \frac{\Theta}{\gamma_{x}}$

and, in fact, N_(θ), will usually be the next integer greater than thisratio of fields of view. However, it could be an even greater integernumber if a larger overlap in the field-of-view of neighboring imagersis desired. Some overlap between the viewing frustums of neighboringdigital cameras is preferred to assist in matching control points aspart of the calibration and stitching operations. Excessive overlap isnot seen as necessary or productive from an engineering or costperspective.

Note that N_(θ) can be an odd or even integer. An even value of N_(θ) ispreferred as it will result in matched left-eye and right-eye digitalcamera pairs, separated by a parallax distance, which more naturallymimics the position of a biologically inspired pair of eyes as the headis rotated side-to-side and back-and-forth. The design implications ofusing an odd N_(θ) value are addressed below.

The parallax distance, p, can be any value from the width of the digitalcamera in use to a value as large as desired. When capturing images foruse by a human, the preferred parallax distance would match theinterpupillary distance of the human end-user or an average humanend-user.

Unfortunately, there is no one interpupillary distance for humans. Table1 shows the results from a study of nearly 4000 humans. There is a widerange of interpupillary distance values, from 52 mm to 78 mm. It'sreasonable to use an average human interpupillary distance of 63.5 mmfor general design and discussion. However, this does suggest that therewould be value in using a mechanism whereby the parallax distance of astereo rig could be tuned to match the interpupillary distance of theuser.

TABLE 1 Interpupillary Distance. Distance between the two pupils on alarge sampling of men and women [Gordon89]. Standard Sample MeanDeviation Minimum Maximum Gender Size [mm] [mm] [mm] [mm] Male 1771 64.73.7 52 78 Female 2205 62.3 3.6 52 76

FIG. 3 is an illustration of the underlying geometry of a general camerasystem design with an even number of evenly spaced mount points inaccordance with an embodiment the present invention. Specifically, itillustrates the ideal geometry 300 for a 360° wide field-of-viewin-plane stereoscopic camera with 6 digital cameras per eye. A desiredparallax distance 301 is achieved by mounting the digital cameras aroundthe parallax circle 302. Since N_(θ)=6, 6 evenly spaced constructiveradial segments 303, 304, 305, 306, 307, and 308 are drawn 60° apart.(Note that because N_(θ) is even, these N_(θ) radial segments could begrouped into N_(θ)/2 co-linear diameter segments of the circle 302.However, for generality, radial segments are used.)

The intersection between each radial segment and the parallax circle 302define N_(θ) idealized “mount points” 309, 310, 311, 312, 313, and 314.The preferred intention is to locate both a left-eye digital camera anda right-eye digital camera adjacent to each other at each of these mountpoints. For example, idealized right eye digital camera 315, withviewing frustrum 317 and idealized left-eye digital camera 316 withviewing frustum 318 are drawn at ideal point 309. (To highlight thegeometry for discussion purposes, the viewing frustums are drawn withthe minimum required 60° of coverage.) Because the digital camera bodiesare considered infinitesimal in this figure, no obscuration issues comeinto play. Thus, the digital cameras are oriented such that theiroptical axes 319 and 320 are in the plane and exactly tangent to theparallax circle 302, pointing in opposite directions. By mountingadjacent digital cameras to these mount points, a minimum sized camerastructure (for a given parallax distance) is realized. This is ideal inthat it results in a minimum distance between the nodal points ofneighboring digital cameras, which will minimize any irregularities inthe stitched image due to parallax.

This mounting configuration would only be possible if the digital camerabodies themselves were infinitesimal. In reality, the digital cameraswill have a finite size and allowances must be provided to dissipateheat and facilitate appropriate electrical interfaces. Thus, the twodigital cameras must be mounted a small operational distance away fromthese mount points. This was illustrated in FIG. 1 and FIG. 2.

FIG. 4 is an illustration of the underlying geometry of a general camerasystem design with an odd number of evenly spaced mount points inaccordance with an embodiment the present invention. It illustrates anarrangement where the number of required digital cameras (per eye),N_(θ), is odd. Specifically, it illustrates the idealized geometry 400of an in-plane design with a 5-camera per eye (10 digital cameras inall) configuration. Defined around the parallax circle 402 are 5constructive radial segments 403, 404, 405, 406, and 407 and 5 mountpoints 409, 410, 411, 412, and 413. Adjacent left- and right-eye digitalcameras can be mounted to these points and as long as the horizontalfield of view of each viewing frustum is greater than or equal to 72°,full 360° horizontal coverage is realized. This is illustrated byshowing the 5 right eye digital camera viewing frustums 414, 415, 416,417, and 418.

The same stitching and blending algorithms could be applied to both theleft-eye and right-eye sets of digital cameras. The difference is that 5unique sets of digital camera pairs cannot be identified. However, ifthe stitching and blending is done in a consistent fashion, it ispossible to overlay the two resulting wide field-of-view images suchthat an immersive stereo effect is realized.

Non-uniform and non-symmetric camera designs are also taught. Forexample, an overall field of view, Θ, less than 360° could be builtwhere not every mount point has two adjacent digital cameras mounted.

It is also possible to realize an architecture with a mix of digitalcameras having different performance specs. For example, any performancespecifications could be varied from digital camera to digital camera,such as resolution, pixel array size, field of view, gain, zoom, dynamicrange or even sensitivity to different regions of the electromagneticspectrum. While these are not preferred embodiments, if called for,these kinds of heterogeneous designs could be realized.

FIG. 24 a is an illustration of the underlying geometry of a generalcamera system design with an odd number of unevenly spaced mount points.It corresponds to an in-plane, 5-camera per eye design. It shows aparallax circle 2402 with five arms 2403, 2404, 2405, 2406, and 2407that are spaced in an uneven fashion. There could be a number ofpossible reasons for such a design, such as using image sensors withdiffering resolution levels or a desire to provide varying resolutionaround the circumference.

There still are five mount points identified at 2409, 2410, 2411, 2412,and 2413. At each of these points, left- and right-eye idealized digitalcameras are mounted as before. But to get the full 360° coverage, theminimum required horizontal field of view for each viewing frustum willdiffer as a function of camera position. FIG. 24 a shows the minimumrequired horizontal field of view for each of the five right-eye viewingfrustums 2414, 2415, 2416, 2417, and 2418 which correspond to the 5right eye digital cameras. The optical axes 2419, 2420, 2421, 2422, and2423 for each digital camera is shown exactly tangent to the parallaxcircle 2402.

By orienting the idealized right eye digital camera at mount point 2409such that its optical axis is tangent to the parallax circle 2402, thein-plane isosceles viewing frustum triangle ABC becomes similar toisosceles triangle OAB, indicating that the field of view angle, γ,2432, required to provide coverage of the scene up to the neighboringdigital camera at mount point 2410, is exactly equal to the angle, α,subtended between the support arms 2403 and 2404. The same geometricrelation holds for all digital cameras. Thus, by using digital cameraswith appropriately sized fields of view, full panoramic coverage can becaptured using such a non-uniform camera design.

FIG. 24 b is an illustration of the same camera system design of FIG. 24a highlighting the left eye viewing frustums to show the complementaryarrangement. Specifically, the complementary left-eye frustums 2454,2455, 2456, 2457, and 2458 and the optical axes 2459, 2460, 2461, 2462,and 2463 are illustrated to show their orientation.

This non-uniform configuration is not the preferred embodiment. However,it is presented to show that such non-uniform designs do qualify aspossible embodiments of the underlying invention.

Elevated Elevation Considerations

Assuming a full 360° horizontal field of view, the fraction of the4π-steradian omni-directional space captured by a particular imagingsolution, centered on a horizontal plane, is given by sin

$\left( \frac{\Phi}{2} \right)$

where Φ is the vertical field of view of the design. For an in-plane,panoramic camera design, the effective vertical field of view, Φ, willapproximately match the field of view of each digital camera, i.e.Φ=γ_(y). For example, a six camera-per-eye Pancam design built aroundOmniVision P/N OV03640-VL9A image sensors would deliver a vertical fieldof view of 50.9 degrees, thus capturing approximately 43% of theomni-directional space.

Extended elevation designs can be realized to increase the fraction ofcoverage. This is achieved by placing imagers in locations that wouldmimic the position of the eyes as the head is tilted forward orbackward. In this case, for a desired vertical field of view, Φ, thenumber of required out-of-plane digital cameras per eye, N_(φ), willsatisfy

$N_{\phi} \geq \frac{\Phi}{\gamma_{y}}$

Again, N_(φ) will usually be the next integer greater than this ratiobut could exceed this value to ensure greater overlap. To ensureimproved high elevation coverage, it may be necessary to use imagesensors with greater horizontal field of view coverage. This would be adecision made on a case-by-case basis.

All of the design considerations described in the last sub-section inconnection with in-plane designs hold for extended elevation designs.The primary difference is that while digital cameras would be locatednear the mount points identified in FIG. 3, FIG. 4, or FIG. 24, theiroptical axes would now be oriented tangent to a parallax cylinder (asopposed to a parallax circle) where the diameter of the cylinder matchesthe desired parallax distance.

In such a design, employing for example, OmniVision P/N OV03640-VL9Aimage sensors, a 360 degree horizontal×140 degree vertical view can berealized using an N_(φ) value of 3, resulting in the coverage of 95% ofthe 4π-steradian omni-directional space.

Extended elevation embodiments are described in more detail below, inconnection with FIG. 11-FIG. 13.

Adjusting Parallax Distance

The parallax distance is an important element governing the performanceof a stereoscopic camera system. By matching the parallax distance tothe interpupillary distance of the end-user, optimal stereoscopicimmersiveness can be realized. However, it has also been widely taughtthat there are situations where using an alternative parallax distancecould be of use. For example, in the field of hyper-stereo imaging, alarger parallax distance is used to introduce greater disparity betweenthe captured images. This can be used when imaging distant objects toenhance the depth effect.

A significant problem in using hyper stereo is that it often enhancesthe apparent depth of objects more than the brain expects, which can bedisconcerting for users. A useful way to compensate for this is toadjust the optics in coordination with the parallax distance adjustment.For example, consider a situation where objects 100 meters away arecaptured with a parallax distance of 127 mm (twice that of an averagehuman). By adjusting the zoom also by a factor of 2, the object willappear with the same size and perceived depth as if it is 50 metersaway. If the user chose to scan around the panoramic scene from such azoomed-in set of cameras, the effect would resemble that seen whenlooking through a set of binoculars and turning one's head. The scenewould spin by at a faster rate, but the brain would be able to processthe scene in a reasonable fashion. These concepts are related to a fieldcalled ‘telephoto-stereo’ imaging, where the zoom factor of the lens isadjusted in coordination with the parallax distance of the camera. Thiscould have enormous utility when viewing public events, like sports orconcerts, in 3D as it would allow the camera to be located at a remoteposition while giving the user a much closer effective point of view.

In a similar fashion, the field of ‘macro-stereo’ imaging makes use ofsmaller than normal parallax distances to enhance the depth perceptionof near field objects. The selection of a parallax distance is soimportant in the field of stereography that a common rule-of-thumbcalled “the 1/30^(th) rule” has been established. It advises that, toget a good stereo effect when imaging a scene, the parallax distancebetween cameras should be equal to or greater than the distance to thenear point of the scene divided by a factor of 30. Since the averageinterpupillary distance of humans is 63.5 mm, this would suggest thatthere could be advantages in using a macro-stereo arrangement whenviewing objects closer than 1.9 m from the camera.

There may be other instances where it is beneficial to adjust theparallax distance dynamically while capturing a scene. This could be ofuse in scenes where the objects are moving or the camera operator wantsto capture a unique visual effect.

Stereo Imaging for Estimating Depth

When the stereo images are, alternatively, fed to a computer (or othercomputational platform), triangulation techniques, that are well knownby anyone versed in the field, can be used to build up a depth map ofthe imaged scene.

The use of stereo images to build up a depth map can be enormouslyhelpful in a number of applications. For example, stereo vision caneasily solve the common computer vision segmentation problem, which isthe ability to separate foreground and background items in a scene. Manymonoscopic image processing algorithms devote significant processingresources to solving this problem before commencing with useful 3Dvision tasks. With stereo imaging, the segmentation problem is solveddirectly, significantly reducing the processing required.

In addition, once the depth analysis of a scene is complete, theinformation could be overlaid with a view of a scene to enhance safetyor operational objectives. And the depth information could be overlaidon either a 3D or 2D view of the scene depending on the interests of theend-user. The ability to add this 3D contextual information to any scenewould have enormous utility in the field of augmented reality.

There is a vast field of computational stereography where algorithmshave been developed in order to back out depth information of a capturedscene by triangulating common points in stereo image pairs. It is wellknown to practitioners in this field that the depth, z, of a point in ascene can be estimated as:

$z = \frac{fb}{\Delta}$

where b is the parallax distance between cameras, f is their focallength and Δ is the disparity, which is the difference in thecoordinates of the point of interest in the two images. The sensitivityof the estimated depth to variations in disparity is given by

${\frac{\partial z}{\partial\Delta}} = \frac{z^{2}}{fb}$

As shown, the parallax distance has a strong effect on the ability toestimate depth. Very large parallax distances can be useful inestimating the depth of distant objects. However, smaller parallaxdistances might be necessary, for near-field performance.

Variable Parallax Pancam Designs

An apparatus that would allow the parallax distance to change withcircumstances will be of great use. Perhaps the simplest example, fromeveryday life, of a mechanism that would enable the required adjustableparallax functionality is an umbrella. A structure could be realizedwhereby the act of sliding a support along a central pole could be usedto adjust the parallax distance of the digital cameras.

Another, slightly more complex mechanism makes use of a rack and piniontransmission. FIG. 5 is a top view of a variable parallax distance,6-arm camera system design, at a minimum parallax distance configurationin accordance with an embodiment the present invention. Similar to thePancam design described in connection with FIG. 2, the variable parallaxdistance imaging system 500 is composed of 12 separate camera daughterboards, distributed around a circle 520 with a diameter 521 matching adesired parallax distance. The six left-eye camera daughter boards areidentified as camera daughter boards 501, 502, 503, 504, 505, and 506and the six right eye camera daughter boards are identified as cameradaughter boards 507, 508, 509, 510, 511, and 512.

As in the idealized designs discussed above, adjacent left- andright-eye camera daughter boards are mounted substantially back-to-backaround each mount point 513, 514, 515, 516, 517, and 518. The locationof each back-to-back set of camera daughter boards is facilitatedthrough the use of the support brackets 519, 520, 521, 522, 523, and524. In the preferred embodiment, the camera daughter boards are locatedas close as possible. However, as is common practice in the field,allowances will be made to allow for adequate thermal relief and toaccommodate the logistics of mounting structure and electricalinterfaces.

A support structure 525 is employed to position the various cameradaughter boards and hold the mechanical transmission used to adjust theconfiguration. The structure is generally arranged as a central portionwith 6 arms extending out in a radial fashion. The structureincorporates a support housing 526, which can be manufactured as onebody or composed of multiple parts that assemble together.

FIG. 6 is an isometric view of a variable parallax distance, 6-armcamera system design, at a minimum parallax distance configuration inaccordance with an embodiment the present invention. It is the samecamera system 500 with most of the same components labeled as in FIG. 5.For clarity, not every component from FIG. 5 is renumbered here. Inorder to support the rack and pinion logistics, the racks engage acentral pinion at three different levels. In the preferred embodiment,the image sensors are positioned in the same horizontal plane. To enablethis, support brackets 519 and 522 include an offset in a downwardfashion. Symmetrically, support brackets 520 and 523 include an offsetin an upward fashion.

FIG. 7 is a isometric section view of a the variable parallax distance,6-arm camera system design, at a minimum parallax distanceconfiguration, highlighting the rack-and-pinion transmission inaccordance with an embodiment the present invention. A pinion gear 553is mounted to a central shaft 550 in a fashion that is common in theart, such that it is fixedly coupled to the shaft and they move as onebody. (A key (not shown) is used in this embodiment.) The shaft issupported above and below the pinion by roller bearings 551. Shaftspacers 552 are also incorporated to ensure proper clearance. Aretaining ring 554 is included to help maintain axial position ofcomponents on the shaft. As the shaft is rotated, the pinion 553 engageseach of the racks 540, 541, 542, 543, 544, and 545 in an identicalfashion.

Inside of each arm of the support housing 526 is a telescoping arm,composed of three elements, an inner, middle and outer element. Eachrack is solidly fixed to the minimum radius end of the inner element ofits corresponding telescoping arm. So, in the figure, rack 542 issolidly fixed to the minimum radius end of the inner telescoping armelement 576. As the pinion turns, it pushes the rack 542 which in turnextends the inner element in an outwardly radial direction. The innerelement is free to slide with respect to the middle element 575 of thetelescoping arm up until it reaches a mechanical limit with the middleelement. Further rotation of the pinion then pushes both the inner andmiddle elements of the telescoping arm outward. The middle element isfree to slide with respect to the outer element 574 of the telescopingarm up until it reaches a mechanical limit with the outer element. Inthis design, the outer element is fixedly mounted to the support housing526 for general support and is not designed to move. The support bracket521 (see FIG. 6) is solidly fixed to the maximum radius end of the innertelescoping arm element 576. Thus, as the rack 542 is adjusted by thepinion 553, the telescoping arms (574, 575, 576) operate to extend theparallax offset of camera daughter boards 503 and 512.

An identical mode of operation occurs between the pinion 553 and eachrack and telescoping arm structure. Rack 545 and the telescopingelements 573, 572, and 571 that connect to support bracket 524 andcamera daughter boards 506 and 509 are also illustrated in FIG. 7.

The above is one possible telescoping arm embodiment. But anytelescoping arm design, as is known in the art, would work. Through thistransmission, the radial position of all camera daughter boards can beextended or retracted in an identical manner and a variable parallaxdistance realized.

Anti-backlash techniques could also be employed to minimize the effectthat excess clearances would have on maintaining accurate radialpositioning of each imager. The shaft could be coupled to a motor, handcrank or any other transmission element that could affect the rotationaloperation of the pinion. These additional transmission elements are notshown but would be obvious to anyone skilled in the art.

FIG. 8 is an isometric view of a variable parallax distance, 6-armcamera system design, at an intermediate parallax distance configurationin accordance with an embodiment the present invention. This is theresulting configuration of the camera design of FIG. 5-FIG. 7 uponoperating the shaft and pinion to extend the parallax distance part-way.In this configuration, the inner and middle elements of each telescopingarm are visible. Thus, camera daughter boards 506 and 509 are supportedby inner telescoping element 573 and middle telescoping element 572.Similarly inner and middle telescoping elements 576 and 575 supportcamera daughter boards 503 and 512. The process is the same for theother four arms such that all of the arms extend in an identical manner,resulting in a uniform change in the parallax distance, p, 521.

FIG. 9 is an isometric view of a variable parallax distance, 6-armcamera system design, at a maximum parallax distance configuration inaccordance with an embodiment the present invention. Similar to FIG. 8,this illustrates the resulting configuration upon rotating the shaft andpinion a maximum amount, such that the telescoping arms are maximallyextended. Again, the inner and middle elements of each telescoping armare clearly exposed and the camera daughter boards enjoy a maximumparallax distance, p, 521.

FIG. 10 is an isometric view of a variable parallax distance, 6-armcamera system design, at a minimum parallax distance configuration withviewing frustums for each of the right eye camera daughter boards andshowing a central electronics component located above the imagingstructure in accordance with an embodiment the present invention. Acentral processing set of electronics is needed to collect the imagestreams from all of the camera daughter boards and stitch together thefinal panoramic 3D result. FIG. 10 illustrates the placement of anelectronics support structure 1020 in proximity to the camera structure.A section cut has been made to show the central processing electronics1021 that would be mounted inside this structure 1020. This structurecan either be connected directly to the camera support structure 526 ormounted separately. In the preferred embodiment, a wired electricalinterface, using serial data transfer, will be routed from each cameradaughter board to the central processing electronics. A separatediscussion is provided below to describe the types of processing tasksthat may be completed on this central processing hub.

FIG. 10 also shows the viewing frustums 1007, 1008, 1009, 1010, 1011,and 1012 that correspond to the 6 right-eye camera daughter boards 507,508, 509, 510, 511, and 512, respectively. As shown, a convenientlocation in space exists just above (or below) the camera structure atwhich the electronic support structure 1020 can be mounted withoutobscuring any of the viewing frustums.

In practice, the shaft can be adjusted in an analog fashion resulting inan operational parallax distance 521 of any value between the minimumand maximum distances allowed by the mechanics of the system. Forexample, embodiments of the invention could be made smaller or largerthan the design illustrated above. The sizes of components used in thisfigure were selected for illustrative purposes only. A much largerversion of this system could be used to enable hyper-stereoapplications. For example, mounting the system to a Naval vessel or amilitary aircraft to better search for and identify threats. Miniaturedesigns could also be realized to adjust the parallax distance at orbelow distances matching the human interpupillary distance. Applicationsof this technology are discussed in more depth below.

A rack-and-pinion transmission was illustrated in the above figure sinceit provides a dramatic illustration of the variable parallax distanceprinciple. However, embodiments of this invention are not restricted togear based designs. For example a fluidic solution (using hydraulics orpneumatics) could be used to control the radial support location. Infact, a fluidic solution might be ideal for designs that require a largechange in parallax distance since it avoids the use of a solidstructure, like a rack, to facilitate a purely mechanical solution. Forexample, the rack lengths in the above design limit the extent to whichthe parallax distance can be adjusted, since the racks can impinge onthe viewing frustums of oppositely mounted camera daughter boards whenat a minimum parallax configuration, as illustrated in FIG. 5. In afluidic solution, hydraulics or gas could be piped in from above orbelow in order to extend the arm length by way of valves and anactuating piston, as would be obvious by one skilled in the art.

Almost any actuation technology could be used to drive the adjustment.This includes linear motors, power screws, magnetic or electrostaticdevices. Active material solutions are also possible. For example, thereare a wide variety of piezoelectric, electrostrictive, magnetostrictive,or ferromagnetic ceramic or shape memory alloy actuation technologiesthat could be used in so-called single stroke, or multi-stroke(inch-worm) type designs. In fact, quite sophisticated position controlschemes, using feedback and feed-forward control techniques, could beemployed to tune the actuated length of each camera arm separately orthe set of arms as a group.

Extended Elevation Variable Parallax Distance

FIG. 11 is a view of an extended elevation camera bracket structure withthree right eye camera daughter boards and three left eye cameradaughter boards in accordance with an embodiment the present invention.FIG. 12 is a zoomed out image of the extended elevation camera armstructure of FIG. 11 that includes wire frame viewing frustums for thecamera daughter boards highlighting the extended vertical field of viewcoverage. This camera bracket mount structure can be used to enableextended elevation embodiments of the variable parallax distance design.A camera support bracket 1119 interfaces with a multi-elevation mountingfixture 1180. This fixture orients three camera daughter boards‘per-eye’ (N_(φ)=3) such that the vertical fields of view of theirviewing frustums overlap by a small amount. The main difference with thein-plane design is that whereas the optical axes were tangent to aparallax circle for the in-plane design, now the optical axes of allcameras are substantially tangent to a ‘parallax cylinder’.

In FIG. 11 and FIG. 12, camera daughter board 1101 has a similarfunction to camera daughter board 501 in FIG. 5. Camera daughter board1101 images the scene through viewing frustum 1181. Camera daughterboard 1201, with viewing frustum 1281, is added to extend the verticalfield of view above the horizon. Camera daughter board 1301, withviewing frustum 1381, is added to extend the vertical field of viewbelow the horizon. In this particular design, the vertical field of viewof each digital camera is approximately 50°. The mounting fixture 1180is designed with 45 degree angled faces to deliver an approximately 5°overlap in the vertical fields of view of neighboring cameras.

In a similar manner, camera daughter boards 1110, 1210 and 1310 providean extended elevation view for the right eye through viewing frustums1190, 1290, and 1390, respectively. Again, the viewing frustums overlapby a small amount to ensure no blind spots and to help with stitchingthe scenes vertically. Such an arrangement allows the realization of anextended vertical field of view at a high resolution and with minimaldistortion.

FIG. 13 is an isometric view of a variable parallax distance, 6-arm,extended elevation camera system design, at an intermediate parallaxdistance configuration in accordance with an embodiment the presentinvention. Wire frame frustums for the three right-eye camera daughterboards at the end of two of the arms are illustrated to highlight howthe frustums cooperate to provide extended elevation performance. Theplatform is shown extended mid-way, but the concept is valid for anyselected parallax distance value. As shown, right eye camera daughterboards 1110, 1210 and 1310 are imaging the scene through viewingfrustums 1190, 1290, and 1390. In a similar fashion, camera daughterboards 1109, 1209 and 1309 are imaging the scene through viewingfrustums 1189, 1289, and 1389.

This multi-elevation design realizes a variable parallax distance in anidentical fashion to that described in the in-plane design. As aninternal pinion is rotated, the arms of the design extend or contractresulting in a variable parallax distance functionality.

Alternate Embodiments 8 Camera Design

FIG. 14 is a top view of a variable parallax distance, 8-arm camerasystem design, at a minimum parallax distance configuration inaccordance with an embodiment the present invention. This figure isincluded to show that embodiments can be realized with an arbitrarynumber of arms. The operative philosophy of the design still governs. Inthe embodiment illustrated, there are 8 left-eye camera daughter boards1401, 1402, 1403, 1404, 1405, 1406, 1407, and 1408 and 8 right-eyecamera daughter boards 1409, 1410, 1411, 1412, 1413, 1414, 1415, and1416. As with the six-camera design, the camera daughter boards arelocated near mount points at the ends of 8 arms through the use oftelescoping components, rack transmissions and support brackets. Forexample, camera daughter boards 1401 and 1413 are mounted to supportbracket 1417, which interfaces to the outer end of a telescoping armmounted to the support housing 1426. The rack 1440 engages with a piniongear which is not visible. Operation proceeds in a substantially similarfashion as that described in the h-armed Pancam embodiment describedabove. An identical mounting architecture is used for the other 7 armsof this design.

Alternate embodiments can be realized that duplicate the multi-elevationdesign described above or that represent wide field-of-view scenes ofless than 360 degrees or even designs where arms of the camera designmay be eliminated for logistical or operational reasons. An example ofthis would be when mounting one of these cameras against the side of abuilding.

Selectable Parallax Distance Pancam Design

There would be significant utility in a camera system able tostereoscopically monitor an entire wide field-of-view area through theuse of multiple parallax distances, simultaneously and without anymoving parts. Such a system would deliver numerous benefits over designsthat rely on moving components to adjust parallax distance. Theseinclude simultaneous capture of the scene, reduced cost and fewermaintenance issues.

FIG. 15 is an isometric view of a selectable parallax distance, 6-armcamera system design in accordance with an embodiment the presentinvention. This Pancam design 1500 arranges three groups of digitalcameras, defined as ‘supersets’, to capture a wide field of view scene,stereoscopically, using three different parallax distances. In thisparticular embodiment, the three parallax distances were chosen to be1×, 2× and 4× the interpupillary distance of a human, 63.5 mm. Thus, thecamera daughter boards are oriented around three constructive,concentric cylinders (not shown) with diameters of 63.5 mm, 127 mm, and254 mm. In practice, any parallax distances and any number of supersetsof digital cameras could be used. For example, the supersets could havethe same parallax distance.

The supersets are mounted through the use of six contoured arms 1521,1522, 1523, 1524, 1525, and 1526. The arms are mounted at one end to acentral hub 1550 and rise along a desired trajectory. At specific radiallocations along the arms, left- and right-eye camera daughter boards areadjacently mounted to each arm by a mounting bracket (these brackets areshown in the figure but not specifically identified with a number).

FIG. 16 is a top view of the selectable parallax distance, 6-arm camerasystem design of FIG. 15. This figure is used to more clearly identifythe camera daughter board supersets in this embodiment.

At a minimum parallax distance, a first superset of cameras is located.Six left-eye camera daughter boards 1501, 1502, 1503, 1504, 1505, and1506 are identified in this superset. Six right-eye camera daughterboards 1507, 1508, 1509, 1510, 1511, and 1512 are similarly identified.

At a second parallax distance, a second superset of cameras is located.Six left-eye camera daughter boards 1601, 1602, 1603, 1604, 1605, and1606 are identified in this superset. Six right-eye camera daughterboards 1607, 1608, 1609, 1610, 1611, and 1612 are similarly identified.

At a maximum parallax distance, a third superset of cameras is located.Six left-eye camera daughter boards 1701, 1702, 1703, 1704, 1705, and1706 are identified in this superset. Six right-eye camera daughterboards 1707, 1708, 1709, 1710, 1711, and 1712 are similarly identified.

FIG. 17 is an isometric section view of a selectable parallax distance,6-arm camera system design in accordance with an embodiment the presentinvention. Constructive wire frame and solid viewing frustums are drawnfor the camera daughter boards of one arm to highlight how, in thepreferred embodiment, the arms are contoured to minimize obscurationbetween camera supersets. Wireframe viewing frustums 1581, 1590, 1781,and 1790 are drawn for the minimum parallax distance camera daughterboards 1501 and 1510 and the maximum parallax distance camera daughterboards 1701 and 1710, respectively. Solid viewing frustums 1681 and 1690are drawn for the intermediate parallax distance camera daughter boards1601 and 1610, respectively. These solid frustums are drawn to highlighthow the contour of the arm is such that none of the camera daughterboards from neighboring supersets of neighboring arms pierce these solidfrustums. For example, camera daughter boards 1702, 1706, 1709, and 1711are preferably positioned so as to not enter the viewing frustums 1681and 1690.

Embodiments that allow obscuration between arms and camera daughterboards of neighboring supersets are possible. In these cases, techniquessimilar to those discussed above for dealing with in-plane obscurationcan be employed to manage the obscuration. Specifically, the user couldbe expected to process out (i.e. ignore) the obscuration. Alternatively,advanced processing could be used to mask out the presence of thesecomponents using feeds from forward cameras, as described before.

As with the variable Pancam design, a central set of electronics willalso be used and preferably mounted in a manner similar to that shownfor the design illustrated in FIG. 10.

Extended elevation versions of the selectable-parallax distance cameracan also be realized by combining a camera daughter board arrangementsimilar to FIG. 11 with an architecture similar to that shown in FIG.15. However, as the vertical field of view of the effective frustums isextended, the vertical separation between the camera supersets will alsopreferably increase to avoid obscuration between cameras fromneighboring supersets. This can lead to longer, more cumbersome andstructurally complex contoured arm members.

FIG. 18 is an isometric view of a selectable parallax distance, 18-armextended elevation camera system design composed of multiple levels ofcamera supersets with different parallax distance values in accordancewith an embodiment the present invention. This design is able to capturea scene from multiple parallax distances, simultaneously, as in thepreviously described design. It also includes the added benefit thateach arm can be mounted to a single vertical base structure at thecenter, which is less complex, structurally. As with the designillustrated in FIG. 15-FIG. 17, there are three supersets of cameras atparallax distances at 1×, 2×, and 4× the average human interpupillarydistance. However, now there are 18 arms instead of 6. In addition, atthe end of each arm, instead of just one left and one right eye cameradaughter board, a camera fixture similar to that illustrated in FIG. 11is used to enable a much greater effective vertical field of view.

All of the arms of the camera are now mounted to a single base pole1850. The minimum, 63.5 mm, parallax distance supersets are supported bysix arms 1821, 1822, 1823, 1824, 1825, and 1826. The intermediate, 127mm, parallax distance supersets are supported by six arms 1921, 1922,1923, 1924, 1925, and 1926. The maximum, 254 mm, parallax distancesupersets are supported by six arms 2021, 2022, 2023, 2024, 2025, and2026.

FIG. 19 is an isometric view of the camera system of FIG. 18 with solidviewing frustums drawn for six minimum parallax distance camera daughterboards to highlight the design placement to minimize obscuration betweencamera supersets in accordance with an embodiment the present invention.As shown, in this extended elevation embodiment, the vertical spacingfor the arms is bigger than for the in-plane design of FIG. 15 to ensurethe camera daughter boards stay out of the viewing frustum of the othersupersets. The most stringent constraint on this is posed by the maximumparallax distance superset, in connection with the viewing frustums ofthe minimum parallax supersets. This is illustrated in the figure, wherethe viewing frustums from the six camera daughter boards at the end ofthe minimum parallax distance arm 1821 are drawn as solid structures.The three left-eye viewing frustums are identified as 1881 a, 1881 b,and 1881 c. The three right-eye viewing frustums are identified as 1890a, 1890 b, and 1990 c. As shown, the length of the pole 1850 and theplacement of the neighboring supersets are preferably chosen to keeptheir camera daughter boards out of the minimum parallax distanceviewing frustums in order to avoid obscuration between supersets.

Summary of Selectable Parallax Distance Benefits

There are significant benefits of simultaneously capturing the samescene from the perspective of multiple parallax distances. Theselectable capability will come into play as a user consumes the videocaptured by the system. Since all cameras are capturing the entire sceneat all times, the user has the capability to choose which parallaxdistance feed to observe.

Specialized configurations of the system could also be developed wherethe optical settings (zoom, aperture, etc) of the various parallaxdistance supersets could be adjusted to give a unique alternative viewof the scene. For example, if a camera system like this was used tocapture a live event, the shorter parallax distance could be used forgeneral observations whereas the larger parallax distance supersetscould be equipped with zoom lenses to provide ‘up-close, zoomed, 3Dimages’ for more immersive views of the action. All feeds could berecorded on a circular buffer and the user would have the capability toselect (and review) the video feed that most effectively captured theunfolding events. This would offer a kind of enhanced ‘instant replay’functionality. A more detailed example of how a system like this mightbe used is described in the Applications section, below.

Other embodiments of this camera can be realized. For example, anysequence of arms could be selected along the pole. Also, as discussedfor the variable parallax distance structure, designs with non-uniformarm spacing and/or designs that make use of digital cameras with avariety of performance specifications can all be considered validembodiments of the invention.

In addition, combinations of the above embodiments are possible. Forexample, a camera system with multiple camera supersets, staticallylocated, in combination with a variable parallax distance design(similar to that described in connection with FIG. 5) could also berealized. A static parallax distance matching the human interpupillarydistance could be used for general observation of a scene while avariable parallax distance superset could be used to selectively zoom inand observe or analyze elements of the scene as desired.

FIG. 25 illustrates a 6-arm, rack and pinion, variable parallax designwith attached drive motor and control electronics in accordance with anembodiment the present invention. This figure illustrates how theparallax distance could be adjusted in concert with the optical settingsof the camera. This kind of arrangement could be used as a stand-alonecamera system or as one of the supersets in the selectable parallaxdesigns described above. The camera system incorporates six arms with 12digital camera daughter boards 2501-2512 (not every daughter board islabeled) in a design identical to that described in relation to FIG.5-FIG. 10. Also shown is a drive motor 2595 that couples to the bottomof the main shaft and pinion gear (both not shown) through a couplinginterface 2596. A central set of electronics 2521 is housed in anelectrical housing 2520. These electronics interface with one or more ofthe camera daughter boards through control wires 2597. In a similarfashion, a motor control interface 2598 is wired between the drive motorand the control electronics. In this embodiment, the control electronicswould operate to adjust the camera operational settings (such as zoomlevel, gain, aperture, etc.) in concert with the drive motor to adjustthe parallax distance to achieve a desired optical effect.

In a following section, a variety of applications are described in whichthe variable parallax distance and/or selectable parallax distancedesigns can enable new levels of performance.

Multi-Camera Image Processing Architectures

One important requirement of nearly all multi-camera imaging arrays isthe use of a relatively powerful image processing platform to synthesizethe usable output image stream. Some designs require more processingthan others.

Data processing for the system proceeds as follows. Data captureinitiates on the image sensor where frames of pixel data are captured atan established sampling frequency and output from the device. After thatpoint, the following processing operations may occur in any particularorder (all of the processing steps are not required to realize a usableimage stream): color reconstruction (often called demosaicing),stitching, blending, compression/decompression, memoryencoding/decoding, as well as other processing steps. In the preferredembodiments, the output of the overall camera system is some form ofdata stored to non-volatile memory and/or images sent to some kind of 2Dor 3D display technology. Each of these processes is described next.

Data Capture

The preferred camera architecture is composed of multiple cameradaughter boards, on which each digital camera operates. These camerascan operate independently. However, to minimize any undesired artifacts(like ‘tearing’) between stitched images of a scene, especially whencapturing scenes with rapidly changing, dynamic content, it ispreferable to synchronize the frame capture of all cameras. For example,if each camera is operating independently at 60 FPS, the frame triggerpoints between any two cameras can differ by as much as 8.3 ms. However,by employing a ‘shutter trigger signal’, initiated by the central boardsand radiating out to each digital camera, the trigger points can besynchronized to within one period of a governing clock signal, which cantypically operate at speeds of 40 MHz or higher. Using techniques commonin the art, frame synchronization of 10's of nanoseconds or faster arepossible, if necessary.

After the pixel data is read from the image sensor, local imageprocessing can optionally be performed on each daughter board prior totransmitting the data to the central hub. Local processing can yieldsignificant benefits in overall system speed and power as it more fullydistributes processing over the entire camera architecture.

Data Transfer Strategies

It is possible to build a single set of integrated electronics thathouse both the individual digital cameras and the central processingcomponents. However, the preferred embodiment employs separate cameradaughter boards to capture and transmit the data to a central hub. Thiskind of architecture is preferred because it is more fault tolerant andmore easily accommodates the swap-out of daughter boards for repair orsystem upgrades.

Many communication standards and formats could be used to transfer thedata, such as wired or wireless transfer and serial or parallel dataformats. Because of the data capture speeds involved, the preferredembodiment would use a wired, serial transfer communication mechanism.Serial data transfer is preferred over parallel transfer because fewerconductors are required (which helps to minimize the size of components)and there is less risk of data corruption due to timing variationsbetween parallel paths. These advantages allow the transfer of data overrelatively long serial paths at much higher frequencies than in parallelimplementations.

Data transfer speeds are of critical importance in multi-camera designs.For example, OmniVision OV02724 image sensors require data transferrates of up to 1600 mega-bits-per-second (Mbps) per sensor. The datarates can be even higher if demosaicing is performed on-board the imagesensor or on the camera daughter board, prior to transmission.Fortunately, serial transmission standards currently exist that caneasily support such data transfer rates, further highlighting thebenefit of employing this communication format.

Electromagnetic interference and other noise artifacts can corrupt thedata transfer process. Techniques common in the art, such as usinginductive filter cores, can be employed to minimize these effects.

Color Processing

For many applications, visible spectrum single-chip color sensors areused. Three components of color at every pixel are realized in thesesensors by locating a patterned color filter, also called a ‘mosaic’,over the pixel array. Often a Bayer color mosaic is used. The advantageis that a single sensor is able to pick-up a full spectrum of visiblecolor. The disadvantage is that an additional demosaicing processingstep is necessary to reconstruct the color information. In the preferredembodiment, the demosaicing process is performed on-board the imagesensor. However, it is also possible to implement the demosaicing in acustom core either on the camera daughter board or on the centralprocessing electronics.

Stitching and Blending

The processing required to stitch corresponding images to create a widefield of view panorama is well known in the art. The primary goal ofthis processing is to geometrically transform the pixels into a commonframe of reference. This is commonly done using a variety oftransformation techniques, projective transformations being one of themost general transforms employed.

Because the digital cameras in this design are mounted in knownlocations, many of the stitching parameters are invariant. However,absolutely precise positioning of the cameras is not required. Whilesome mechanical adjustment of the camera daughter boards is advised,fine tuning of the stitch parameters can be achieved by calibrating thecamera before use or even in real-time by performing periodiccalibrations in parallel with standard operation.

Calibration is often facilitated by using a patterned target, sinceknowledge of a geometric pattern can be used to align the views fromneighboring cameras. This can be achieved by either placing a physicaltarget in front of the camera or by projecting a pattern of structuredlight. For real-time recalibration, the latter technique is preferred.For visible spectrum sensors, this light could even be concentrated inthe near infrared region of the spectrum, to be invisible to the humaneye but visible to the image sensor (since many visible spectrum sensorsare also sensitive to the near infrared spectral region). Projecting astructured light pattern can also be leveraged to perform more accuratedepth analysis of objects in the scene.

Stitching performance will be affected by the distance of objects fromthe camera. (This effect will increase in severity as the distance toobjects decreases.) This can be addressed through calibration steps ormanual adjustment of stitching parameters by the user. In the idealcase, the stitching algorithm would take into account the distance ofobjects in the scene. A 3D estimate could be attached to every pixel ofa scene enabling the use of depth dependent stitching parameters. Thiswill result in a dramatically improved stitched result.

Besides stitching the various images together there are additionalprocessing steps that are necessary in order to create a plausibleoutput image. In particular, it is common, due to differences in theoperation of the digital cameras, to see distinct differences (in color,contrast, etc.) in the regions of the bare stitched image correspondingto the different image sensors. This can be especially apparent if thereis a significant difference in the brightness of different areas of acaptured scene (for example, when capturing a normally lit interiorroom, with windows looking out onto a bright sunny day). Due tolimitations in the dynamic range of typical image sensors, these kindsof common imaging scenarios make it challenging to create a plausiblefinal stitched result. Accordingly, an additional processing step isindicated to identify and blend these transitions to make the overallresulting image more continuous. Such blending techniques are commonlyknown in the art.

Codecs

Due to the significant volume of data that can accumulate in videoapplications, a preferred embodiment would integrate a video compressionstage into the data processing framework. This could be employed forreal-time processing or as the data is written to either volatile ornon-volatile memory. There is a wide variety ofcompression/decompression (CODEC) schemes that can be employed for imagestreams, but H.264 and MPEG4 are two common standards used currently.The use of a CODEC can decrease the amount of bandwidth and memoryrequired to transmit and store the image streams by a factor of up to 50or more. The actual compression amount will be a function of the amountof motion in the scene, the accepted level of ‘loss’, and theeffectiveness of the compression scheme, as is known to those skilled inthe art of video compression and decompression.

Memory Encoding/Decoding

Both volatile and non-volatile memory solutions are required foroperation. Any realistic memory architecture can be employed. Ingeneral, a wide area of volatile memory is typically reserved as a framebuffer to store multiple sequential frames of video for each digitalcamera. The data can be compressed, uncompressed, stitched or unstitchedor stored in any convenient form.

In a similar way, non-volatile memory can be employed to realize digitalvideo recording (DVR) functionality or simply to hold a large volume ofrapidly accumulating data from individual digital cameras prior to beingcombined into a final processed image stream in a post-processing (orparallel-processing) step.

The critical factor in interfacing with memory is the memory busbandwidth. For example, consider the case of a camera system builtaround the OmniVision OV02724 sensor, identified above. In anarchitecture that transfers all 12 camera feeds onto one memory bus, aone-way memory bandwidth of 19.2 Gbps would be required. And, in fact, atotal memory bandwidth much greater than this level is actuallyrequired, when considering that once written to memory, the data mustalso be available for additional processing operations (stitching,blending, etc.) and export for real-time display or storage innon-volatile memory.

There exist many useful techniques employed by those skilled in the artto optimize memory bandwidth usage. One example involves a situationwhere two users want to tap into the same video stream in memory at twotimes, a few minutes apart. Instead of instituting a new memorytransaction for each user, the data can be pulled out once and then atime-delay circuit (such as a barrel shifter) could be implemented todelay the video for delivery to the 2nd user.

In the case of wide field-of-view imaging where smaller cropped sectionsof the video is sent to individual users, the situation is morecomplicated since any two viewers would likely not want to view theexact same sections of memory. However, optimization schemes could bedeveloped that used statistical techniques to grab and time-delay largeparts of the ‘popular’ sections of video and then fill in the extrasections as needed.

And while optimized memory interface techniques can be employed, thefact remains that multi-camera systems have significant memory bandwidthrequirements and advanced memory architectures will be required toaddress this need. A preferred embodiment is to use an AdvancedeXtensible Interface (AXI) bus structure. An AXI bus structure ispreferred because it not only can support the high bandwidthcommunication required but also because it is compatible with a widevariety of commercial cores (such as ARM cores) that can be leveraged tocomplete the processing tasks.

Video Output

Implementing an effective display mechanism for the captured andstitched video is a natural requirement. The video can be formatted fordisplay on either a 2D or 3D display technology. For a stereo visionsystem, it is most natural to consider the use of 3D displaytechnologies such as head mounted displays and 3D TVs. Anothertechnology of interest is a so-called Immersive Display Experience′taught by Perez in US Patent Application #2012/0223885. This technologycan be thought of as a kind of ‘immersive cave’ where 3D displayssurround a user to give a fully immersive experience. All of these 3Ddisplay technologies work by displaying separate image feeds to eacheye. Often, for a 3D display, a specialized output format, such asside-by-side, top-bottom or ‘full HD 3D’, is required. Different corescan be written to enable the appropriate output compatibility.

Users will sometimes not want to view the captured image streams in 3Dformat. In these cases, 2D image stream displays of the output from oneparticular ‘eye’ can be displayed. However the 3D capabilities of thetechnology can still be leveraged by, for instance, overlaying 3D depthinformation of different objects in the scene, the depth estimateshaving been estimated through stereo triangulation, as described next.

Additional Processing Steps

A variety of additional processing steps can optionally be performed onthe data. The most likely additional processing step is to performstereo triangulation to estimate the depth of all objects in the scene.Depth information can be very useful for a number of applications as wasdiscussed earlier in this document.

Additional sensor technologies could be added to the platform to augmentperformance. For instance, time-of-flight emitters and sensors could beadded to enhance depth estimations of the system.

More significantly, orientation tracking technologies likeaccelerometers, gyroscopes and magnets, similar to those used in thehead tracking systems of commercially available head mounted displays,could be used to sense the orientation of the camera system during imagecapture. An application discussing the use of such technologies tocontrol memory encoding operations is described later in this document.

Processing Architecture

One of the most significant challenges in realizing a processingarchitecture is satisfying the bandwidth requirements of a typicalstereo processing algorithm. The system must support one way, singlechannel bandwidths on the order of 1.6 Gbps and aggregate bandwidthswell in excess of 20 Gbps (perhaps even exceeding 100 Gbps). While thesedata processing rates are significant, they are not insurmountable,especially if creative processing architectures are employed includingdata compression schemes, parallel processing and separate memoryinterface designs.

In addition to overall data transfer speed, the architecture must alsoincorporate technologies that facilitate high performance projectiveimage processing operations. For this, there are a number of digitalprocessing technologies that can be employed. These include standardsingle- and multi-core CPUs, digital signal processors, graphicalprocessing units (GPUs), field programmable gate arrays (FPGAs), ASICsor any other electronic processing device.

GPU technology, in particular, has grown in capability in recent yearsto provide just the kind of floating point parallel processing thatwould be needed to solve many of the processing and memory interfacechallenges involved.

FPGAs are used extensively in the field because of the numerous benefitsthat these components provide, including: Scalability, Ability toperform massively parallel processing tasks, flexibility, andreconfigurability.

Each processing technology offers a unique capability that can beleveraged to address the image processing requirements. Hybridarchitectures could therefore be employed to combine complementarycapabilities. For example, some floating point operations, likecalibration, stitching or blending, could be performed on a GPU, whilehigh speed, fixed point operations could operate on the FPGA fabric.There is a lot of flexibility that can be applied. And the relentlesspace at which processing capability is growing, as predicted by Moore'slaw, will continue to ease the task of designing a comprehensiveprocessing solution.

Applications

It is clear that there are numerous applications to which an effectivewide field-of-view stereoscopic imaging technology can be applied.Adding the adjustable or selectable parallax distance functionalitydescribed herein will further enhance potential performance. Thefollowing sub-sections describe some general and specific applicationsthat are enabled through the use of various embodiments of theinvention.

General Enabled Applications

The following describes the way in which this technology will enhancenumerous applications:

Telerobotics: When remotely operating robotic platforms, like UAVs,USVs, and UGVs, humans rely almost entirely on visual cues. Sometimesthese are provided through a single monoscopic camera (as is the case inmany Unmanned Aerial Vehicles), or through a single pair of stereocameras. The wide field-of-view, high resolution, stereoscopiccapabilities of the underlying technology address many of thelimitations inherent in tele-robotic operation. Integrating now theability to adjust parallax distance provides at least two additionalbenefits: First, the parallax distance of the capturing camera systemcan be fine-tuned to the interpupillary distance of the operatorenhancing the operator/platform immersive link. Second, the parallaxdistance can be adjusted to enhance 3D depth estimates which can beoverlaid in the display to provide additional context for the operator.

Surveillance: The underlying technology is ideally suited for use as asurveillance node due to its ability to see in all directions at alltimes. And the stereoscopic capabilities can be leveraged to build up adepth-map of a scene, which can be used to more effectively trackobjects and/or perform things like facial recognition. The adjustableparallax distance embodiment can be leveraged to enhance these depthestimates. For example, consider mounting a large scale version of thecamera system described herein to the deck of a Naval vessel or to thefuselage of a military aircraft. The parallax distance could be adjusted(possibly in an autonomous manner) to better identify and track threatsin different environments.

Traffic Control: A 360 degree imaging solution is critical for trafficcontrol, not only to capture the presence of on-coming automobiles downintersecting streets, but also to identify pedestrians in any directionand anticipate their desire to cross. Such a system could be used tooptimize the flow of traffic and pedestrians. In addition to enhancingsafety, such systems can provide dramatic savings in commutertravel-time, automobile starts and stops (which are associated withmaintenance requirements), fuel consumption and emissions. Estimatingthe distance and velocity of objects in such a scene is criticallyimportant to estimating closing distance of all parties and optimizingflow and safety. A variable baseline design could be adjusted based onthe distance and velocity of on-coming traffic to improve performance. Aselectable baseline system could be used to more effectively estimatethe distance and velocity of both near and distant traffic.

Collision Avoidance: If such a system was attached to a vehicle(automobile, marine vehicle, or aircraft), it could be used to warn ofan impending collision with another object (another vehicle or animal)from any direction (like a deer jumping out of the woods). The variableparallax distance would allow the 3D capabilities to be exploited over awide range of distances. This could have perhaps the most significantimpact for aircraft since there are both so many blind spots and solittle time to react given the closing speeds involved.

Sports/Entertainment Event Viewing: The capabilities of the underlyingtechnology can immerse a user in a scene. This can enable more effectivecapture and broadcast of events with a perspective that is not possiblewith current technologies. Below, a detailed discussion is provided onhow the technology described herein can enable capture of live events ina new and innovative way.

3D estimates of captured scenes could also be used to add specialeffects to a scene by improving the ‘tracking’ of any captured scene. Asthose skilled in the art of Computer Generated Imagery (CGI) are aware,good tracking is critical for adding fabricated elements to scenes in arealistic and size-accurate way.

Mapping and Surveying: The technology could be used to capture the 3Dmeasurements of any space of interest. 3D measurements can also be fedto an overall mapping operation, such that a solid model of the scene isconstructed, using the camera as a reference point. The large volume offrame data can be continually averaged and combined statistically toimprove the fidelity of the model. Furthermore, if the camera is mountedto a moving platform, motion tracking can also be employed and fed intothe modeling procedure to expand the overall model and further improveaccuracy. And if particular elements of the scene must be captured withincreased precision, the variable baseline of the system can be adjusted(either manually or autonomously) to fill in the required details. Suchmeasurements can be fused with technologies, such as time-of-flightemitters and sensors to realize a precise understanding of the depth ofobjects in the scene.

This could be used in the field of interior design and renovations toestimate the length, area and volume of a space as designers work torenovate or decorate an area. In a similar fashion, integrating a cameralike this on a UAV and sending it on a mission to autonomously inspectpipelines, power wires, bridges or roads could represent a significantsavings in manpower from having to send a human to do this monotonousjob.

The estimated depth could also be leveraged by applications such asaugmented reality where computer generated content is overlaid with thescene to improve overall performance.

Telepresence: The immersiveness of this system would allow users toexperience and interact with a remote scene and people at anunprecedented level. One particular application where this could beparticularly useful is for doctors to visit patients in remote or hardto reach hospitals. Remotely, the doctor would have the capability tosee the patient, the instruments to which the patient is connected andeven interact with the nurses and family members in the room. Theimmersiveness from the high resolution and depth provided by the systemwould deliver a greater level of perception to allow the doctor to moreeffectively do their job. The adjustable parallax distance capabilitywould allow the remote user to more effectively discern details of thescene.

Remote Tourism: The camera system could be used to capture scenes ofinterest. For example, hotels, restaurants or other public venues ofinterest could be captured for commercial promotion. Homes could becaptured for real estate sales. Live feeds could be used to captureevents at public venues. The selectable- or variable-parallax distancecould be automated or expertly adjusted to deliver scenes with optimaleffect.

Thrill Rides, Gaming, and Training: The system could be used to captureevents and scenes to be played back to users on demand. For example, thesensation of standing at home plate in Fenway Park when receiving a 100MPH fast ball during the World Series could be captured. Such scenescould be played back for interested fans at home or in museums to allowpeople to experience events from a unique perspective. In the same way,the system can be used to more effectively train people for criticalsituations in the most immersive way possible. For example, soldierscould be trained on what to expect in combat by playing back video froma helmet mounted version of one of these camera systems. The variableparallax distance of the system could be adjusted based on what scenariois being captured or perhaps continuously as the mission is conducted.

Live Event Capture—Memory Creation

One of the most compelling uses of the technology described herein is incapturing and redisplaying live events, like sporting or concert events,in an entirely new way. These events could be professional or amateurevents or even general public assemblies or occurrences.

The opportunity is best explained by considering a common scenario:Parents take great pleasure as spectators at their children's schoolsports or concert events and they often try to capture these by holdingup their phone-camera or a regular camera during the proceedings. Andyet this process is almost self-defeating since that parent is usuallyso busy trying to capture the scene and hold the camera steady that theycan't enjoy the moment itself. In addition, since they rarely know when‘the good parts’ will occur, they often capture long periods ofuninteresting footage and have little time for post-process editingafter the event. The cameras in use are often of low quality and at poorvantage points to capture the action. And, finally, if they arecapturing the scene in 3D, they may be too far from the action to get agood 3D ‘effect’.

The camera system described herein could be coupled with a video capturesystem that could overcome all of the above problems and create an eventcapture system of dramatic usefulness not only for parents but for anyspectator of any event. In fact, the following idea would work whetherthe underlying camera system was a monoscopic or stereoscopic widefield-of-view camera system. In the following, the more complexstereoscopic embodiment is described in order to further illustrate howthe adjustable baseline embodiments of the invention could be leveragedto enhance the performance of this system. The preferred embodimentwould operate as follows:

FIG. 20 is an isometric view of a typical event where a spectator mightwant to capture video clips from a selectable parallax distance widefield-of-view video network in accordance with an embodiment the presentinvention. This particular image is of a typical child's soccer game:

Four of the extended elevation, selectable parallax distance cameras2051, 2052, 2053, and 2054 are mounted adjacent to a soccer field 2050.The cameras can be set-up at a convenient location, such as the two25-yard lines on either side of the field. The cameras could beconnected in a wired or wireless data network architecture. Integral tothe system would be the use of a central node to control the entiresystem.

These cameras would not even have to capture a full 360 degree panorama,but perhaps a 180 or 200 degree horizontal field of view. In thisparticular embodiment, each of the cameras would be designed to cover a200 degree horizontal×100 degree vertical field of view. A constructivewire-frame capture frustum 2094 for camera 2054 is illustrated in thefigure. Identical frustums exist for the other three cameras.

The cameras could be elevated on a stable platform to ensure a good viewof the field. In this case, a 3 meter pole is used to mount each camera.

As the game progresses, the cameras would record a high resolution widefield-of-view image of the scene on a circular buffer that getsoverwritten after a certain amount of time.

For a typical spectator 2060 (such as a parent), for most of the game,nothing of interest would occur and they would be able to relax andenjoy the event.

However, if something remarkable occurs such as their daughter scoring agoal, after the celebration, that parent 2060 would then be able to pullout their phone, tablet or other smart device and interface with thecamera network at the field in order to capture the action 2070 thatjust occurred.

In an embodiment, a software application would operate on their devicethat would interface with the central camera system node. This node mustbe able to initiate separate threads for each user that will want toconnect and it must also implement logic for multiplexing the requestsfor video so that no requests are lost. It will also have to optimizememory interactions to preserve bandwidth using techniques similar tothose described earlier in this document.

FIG. 21 is an illustration of a graphical user interface (GUI) 2100 thatcould be used to engage with a selectable parallax distance widefield-of-view video network for customized video clip purchase inaccordance with an embodiment the present invention. This interfacewould be part of a software application operating on a particular user'sdevice. It will represent a portal by which the user can specify thevideo clip of interest they would like to obtain. Preferably this portalwould provide the most convenient and intuitive interface possible sothat end-users could quickly make accurate performance selections.

In reality, all of the information shown in the figure might notcomfortably fit on one screen. In this case, the interface could belogically broken into multiple interface screens. The parent/user wouldideally be able to make the following selections:

Cropped View: The GUI would display a representation of the field 2150and the cameras 2151, 2152, 2153, and 2154 in place at the field. Theuser could select from which of these cameras to grab footage (multiplecameras could be selected). In addition, they would be able to adjustthe viewing frustums 2171, 2172, 2173, and/or 2174 to define how wide(or how high) an angle to capture (for example, do they want the entire180 degrees of capture or perhaps a limited range of azimuths?).Snapshots from each of the cameras could be provided in preview windows2161, 2162, 2163, and 2164 to assist in setting these crop settings.This process can be thought of as the user specifying their desired‘capture frustum’.

Parallax Distance: For general viewing, the camera superset with aparallax distance matching the human eye could be used. However, forclose-up shots, a longer parallax distance in concert with a greaterzoom could provide an effectively closer shot of the action. Thus, theuser could set the parallax distance 2101 from which to pull video data.For example, this might be specified as a multiple of a human'sinterpupillary distance (IPD). Alternatively, a zoom factor 2102 may beselected. Since the system will operate on already recorded content, theselection of parallax distance and zoom will likely be linked and willprobably be selectable from a drop-down list of possible alternatives.For example, options may be presented such as ‘view 2× closer’, ‘view 4×closer’, etc. where the parallax distance and zoom settings are linked.

Quality: Resolution 2103, frame-rate 2104, compression-quality 2105 andoutput format 2106 could all be specified. Advanced settings 2107 couldbe accessed on additional pages.

Clip Length: The user would specify the time period of the desired clip.This could be as simple as saying ‘give me the last 5 minutes’. Or moresophisticated editing might occur. One possible embodiment would use atime rule 2110, where the sliders would be used to specify thestart-time 2108 and end-time 2109 for the clip of interest. In thepreferred embodiment, sample video of the clip would be displayed in thepreview windows 2161, 2162, 2163, and/or 2164 to assist the user inspecifying the start/stop times. However, if bandwidth andcommunications are constrained, periodic snapshots of the action couldbe provided to assist the user in selecting a start and end point ofselection. Because of the circular buffer, the users would be able toselect for off-loading video from as far back as, approximately, thelength of the circular buffer.

In some circumstances, the scene might be static and a parent's ‘idealperformance settings’ may be known before the event starts. One examplewhere this might be the case is viewing a child's holiday concert wherethe parent has a good idea of where the child will be located within theviewing frustum of the camera. In cases like this, the parent couldcreate ‘pre-set performance settings’ on the network camera system.Then, at exceptional moments in the proceedings, they could capture thelast few moments by simply tapping an indicator like ‘purchase the last2 minutes’ or some other customized setting to enable quick clipcapture. This would further alleviate the problem of diverting aperson's attention from enjoying the event of interest.

Once all the selections were made, the user would ‘purchase’ the clipfor a reported price (or in exchange for receiving some kind ofsponsored content, like an advertisement) and, based on a user profilethat would already be set-up on the system, the purchased clip would beelectronically delivered to that user for future download at theirleisure.

The system would be designed to allow a large number of users tointerface with the camera network in a multiplexed manner to ensuresatisfactory capture of customized video clips of a scene for all users.

There are numerous benefits that this new process would provide. Themost significant is the following:

Preserving the Memory: This system would solve the ironic problemencountered by many parents that show up to watch their kids' sportingor performance events; namely, the practice of holding up a low qualitycell phone camera to capture an event because the parent ‘wants tocapture it as a memory’ when, in fact, the process of trying to capturethe video destroys that parent's ability to enjoy the moment and form alasting memory of their own.

The other benefits are as follows:

Best Seat in the House: Because only one or a few of these cameras areneeded, prime locations can be selected to capture the best view of theevent. Since each person can tap into that video, everyone would havethe option to capture quality video from a great location.

Real Time Editing: This would solve a problem for parents that try tocapture the entire event with the idea that they will go back later andpull out the important clips, which can occupy hours of their valuabletime. With this set-up, the moments that most resonate for each parentwill motivate the capture, resulting in real-time editing of the event.At the end, each parent will have, waiting for them, just the clips ofthe experience that they most want to remember.

High Quality Camera: Because operators are not selling the camera butthe ‘Video Camera As A Service’ (VCAAS), the business model supports theincorporation of much higher quality cameras and optics than wouldnormally be used by general spectators. Thus, high resolution, highframe rate, wide dynamic range and/or superior color depth images couldbe captured and the user could select the quality of image to off-load.The circular buffer length could also be significant. In many cases a 10or 15 minute buffer would be adequate since most spectators know almostimmediately if they want to capture something that just happened.However, if the market called for it, longer buffers that capture theentire event could also be realized.

Stitching in Post: The process would also support operations where theseparate video streams could be off-loaded in parallel and stitched ‘inpost’. This would eliminate the stitching and blending steps from actingas processing bottlenecks, especially if very high resolution and/orhigh frame rate video was captured.

Again, the above system would work for either traditional monoscopic orstereoscopic capture of a scene. In the preferred embodiment,telephoto-stereo techniques would be used to coordinate the parallaxdistance with standard camera control parameters, like zoom factor,aperture, gain, etc., to create a more effective immersive capture ofthe scene. For example, one set of cameras might be set-up to match theinterpupillary spacing of a human while one or more additional supersetsof cameras could be set-up with alternate parallax distances. A mainview of the scene would be captured by the 1^(st) set of cameras whereasthe additional sets could be used to realize an enhanced view ofinterest.

A hybrid camera embodiment combining a variable parallax superset on topof a selectable parallax distance design could be used. Such a systemcould use a combination of depth measurements and image processingtechniques, like optical flow, to ‘follow the action’. In doing so, thevariable parallax stage could be adjusted in tandem with the camera zoomfor more effective immersive 3D capture of the events, as was discussedin connection with FIG. 25, above. In the preferred embodiment thiswould be done autonomously. Combining such a camera with a selectableparallax distance matching the IPD of a human, could offer superiorperformance.

Such scenes could be captured for general enjoyment or 3D analysis. Forexample, a coach might want to quantify the position of players on thefield for analysis. The feeds from different parallax distance captureswould offer a variation in the accuracy and range of the depthestimates.

As for delivery, in this embodiment, the video clips are electronicallydelivered to become part of the users overall media library. There aremultiple modalities by which the user could enjoy the custom createdcontent. In one example, camera operators could work out a deal withinternet service providers, cable TV companies or any group thatdelivers media for consumption by end-users, like Facebook, to allow thevideo clips to be accessed directly through the consumer's cable box orhome entertainment network. In a preferred embodiment, while at homewatching television, they could bring up the guide (often referred to asthe Electronic Program Guide (EPG)) on screen and a new tab of personalvideo clips could conceivably be displayed. The parent could immediatelycome home and share clips of their child's amazing soccer play withfriends and family. Short trailers of each clip could also beautonomously created, perhaps by selecting the few seconds with the mostaction in the clip, to assist the user in selecting a clip. Thefunctionality can be categorized as a method to save and view video thatis part of an overall private media collection. Cloud and locally-basedmedia management systems are envisioned for organizing the video clipsin a manner identical as if the clips were captured with a personalcamcorder and edited in a more traditional way.

Tapping into video as a function of global position is also envisioned.In such a case, the video, flagged for public release by the owner couldalso be tagged with the global position of where it was captured, usingtechniques common in the art.

There may be privacy concerns in using a system like this. However, thefact is that at most public gatherings, any person is able to capturevideo of an event. Indeed, many people do, as discussed above. Thecamera system described here, however, may actually provide more controlover privacy. Since each person would have to register with the cameranetwork, presumably with their own account, there could be ways tocontrol who is granted access to the video data and who owns the videoclips after purchase. For example, at a school event, mechanisms couldbe put in place to allow only authorized family members or friends tolog-in and download content.

Policies can be established to clarify ownership of purchased videoclips as part of the user registration process. For example, thepolicies may guarantee that once a particular clip (created as afunction of a number of user-selected performance settings) is deliveredelectronically, it is subsequently destroyed. In most cases, thepurchaser would view that video at home with family and friends. Butpreferably they, too, could be allowed to add a protective code to thevideo clip in the case that they'd want to release it for publicconsumption. There are many options.

Control over the capture, sale, and ownership of video clips can extendto the use of this system at commercial events. For example, this systemcould be employed at a concert event and the video sold to concert-goersfor an additional price. To guard against piracy, the video clips couldbe encoded with a protective code (similar to the way commercial BluRaydiscs are protected), with information like copyright data, watermark,event ID, camera location and viewing vector information, to preventunauthorized sharing or public release.

Using the content for advertising can be quite useful. In some of theprior art, significant effort is expended to study marketing trendsrelated to what video content was selected and viewed by consumers. Thiscamera system offers an even more in-depth view of marketing trends.This camera system allows not only the ability to understand how manyconsumers choose to watch a particular commercial event, but also thestatistical data (perhaps even broken down by demographics) on whatscenes (and viewing vectors) within that video are being selected.Information like this can significantly enhance targeted marketingefforts.

Viewing vector specification may also occur on the user side. Forexample, it's possible that special set-top boxes (STBs) would bedeveloped that would receive the entire wide field-of-view video feed.These STBs would allow a variety of viewing modalities. For example, ifa user wanted to experience the scene with a head-mounted display, theSTB would send a cropped area of the scene but also respond to the headtracking cues from the HMD, adjusting the view pointer to allow the userto ‘look around’ the event. The end-user could in theory also interfacethe STB to an ‘immersive cave’ type display system, a room equipped with360° or even omni-directional 3D screens, allowing the display of theentire video stream. In the former case, data on what parts of the eventthat the end-user focused could be streamed back to the source in orderto gather market trends. This is similar to some of the prior art whereusers' viewing habits are tracked.

Processing in Post

In embodiments of the invention, the processing architectures will bepowerful enough to perform all required processing and export the videoin real-time. This would allow its use in a number of live videoapplications, such as live sports capture or surveillance. However, inmany situations, as in the soccer application described above, it maynot be possible (due to bandwidth limitations of the processingarchitecture) or even necessary to perform the image stitching, blendingor other processing steps (such as depth computations) in real-time. Inthis case, a realistic strategy is to write each of the separate imagestreams to non-volatile memory in real-time, while processing the scenein a slower, parallel, process.

As discussed above, because video data can grow significantly, it'slikely that a compression core would be integrated into each of thecamera feeds to dramatically reduce the amount of non-volatile memoryrequired to store the image stream. In this preferred embodiment, theseparate camera streams would be written to memory in a compressedfashion. The processor, according to its own schedule, would recall eachframe of the image stream, build up the full frame rate stitched andblended image stream, perform any required additional processing, andre-write it to non-volatile memory (perhaps with additional compressionapplied). Users would then be able to pull up the stitched and blendedomni-directional video for viewing a short time after the capture of theoriginal scene. In the preferred embodiment, this processing would occurautomatically in tandem with video capture and be largely transparent tothe user.

FIG. 22 is an illustration of a dynamic event where an extendedelevation, wide field-of-view video system could be used to capture anevent, but where the camera orientation is dynamically changing inaccordance with an embodiment the present invention. This represents ascenario where post-processing of video would work well. A skier 2201would mount a Pancam 2202 to his helmet. The central processingelectronics and memory may be too large to also mount to the helmet butcould be integrated into a vest (or some other wearable component).Serialized data transfer from the camera to the vest would proceedexactly as described above. An extended elevation design is preferredthat would provide a capture frustum of 360 degrees horizontally and 100degrees in elevation. In contrast to single point of view cameras, asthe skier progresses down the mountain 2200, the camera will captureevents in all directions, such as the antics of other skiers 2211 and2221 on the run.

After the run, perhaps on the way back into the lodge or while standingin a lift-line, the electronics would post-process and stitch thecaptured video frames. At a future time, the skier could play back theentire run or series of runs to their friends and family through a 3Ddisplay technology like the Oculus Rift head mounted display. Thistechnology would rival the GoPro® cameras that are being used in a hugevariety of activities around the world today.

Eventually, once processing electronics become powerful enough andaffordable enough, this stitching can happen in real time so that notonly will, for example, that skier be able to capture each run forfuture viewing, but they might also be able to transmit real-time videoto interested end-users who can view the ski-runs as they happen.

Motion Controlled Memory Encoding

The above application highlights a significant limitation in mounting atraditional, single point-of-view, camera to a helmet (or other movingstructure) to capture a dynamic event. This could be during a ski orsnow-board run, while surfing, biking, parachuting or performing anykind of dynamic activity. Users of these systems often complain aboutthe need to hold their head (or the moving structure) steady in order tocapture video of a particular event. The technology proposed hereinsignificantly alleviates this capture requirement because of the widefield-of-view captured.

While this solution will capture a much larger fraction of the overallscene, the one issue that will still be present, however, is thatbecause the camera platform continues to move during capture, playbackof the raw video will be unsteady, with an orientation matching that ofthe platform during capture.

This problem can be addressed using motion tracking technology. Inrecent years, there has been significant progress in developing highperformance motion tracking with minimum latency at affordable costs. Ofnote is the head tracking technology in recently developed head mounteddisplay technologies, such as the Oculus Rift, made by the OculusVRcompany. The motion tracking technology typically uses a combination ofaccelerometers, gyroscopes and magnets to estimate platform positionwith minimal latency and zero drift.

The output from technologies like this could be leveraged for use inencoding the video as part of an embodiment of the invention. Forexample, in the skiing application, a wide field of view camera system,combined with head tracking sensors, would be mounted to the ski helmet.In the preferred embodiment, the sensors would, in real-time, measureall three rotational orientations of the head: roll (tilting headside-to-side), pitch (tilting head back and forth), and yaw (rotatingthe head side-to-side around an axis substantially aligned with theneck). (Three translation degrees of freedom could also be estimated asadditional degrees of freedom.) For each frame of the image streamcaptured, this tracking data would be written directly into the datastream as a series of three (or six), 4-byte words, during the blankingtime that is normally present at the beginning of the frame (e.g. duringthe ‘vsync’ time period).

FIG. 23 is an illustration of the encoding-to and read-back from memoryof a cropped region of interest from a camera feed where the capturecamera orientation is dynamically changing in accordance with anembodiment the present invention. It is used to illustrate potentialstrategies that can be implemented to ‘subtract out’ base camera motionto allow a user to more easily control the viewing vector for playback.This figure illustrates two sequential frames of image data in memory,each represented as a rectangular area in memory. These are two framebuffers that contain all of captured pixels of that wide field-of-viewframe. At one instance in time, wide field-of-view frame 2311 is storedin this frame buffer. At the same time, a user desires to observe acropped section 2312 of this overall memory array. The cropped sectionrepresents a particular viewpoint or object that the user would like toobserve. In the image, the viewer is focusing on a tree with the sun inthe upper right. The cropped section is characterized by a horizontalnumber of pixels 2302 and a vertical number of pixels 2301. The upperleft corner of the cropped region is offset from the upper left cornerof the frame buffer by Δ_(x) _(i) pixels horizontally 2313 and Δ_(y)_(i) pixels vertically 2314.

The cropped view starts out aligned with the overall frame buffer array.In the next frame of video captured, however, the camera orientation isexpected to vary due to, for example, the roll, θ, pitch, φ, and yaw, γ,of the skier's head. For example, if, in the time between capturing thefirst and second frames of video, the skier's head pitches up, rollsright, and yaws left, the cropped view 2322 would have to changeorientation in the second frame buffer 2321 to maintain the generalorientation of the scene of interest. Now the offset distances are Δ_(x)_(i+1) 2323 horizontally and Δ_(y) _(i+1) 2322 vertically. Moresignificantly, the desired area of memory to view 2322 is now at anangle. So, in the proposed memory interface method, the processor wouldbe programmed to account for the encoded motion to playback a smoothstream of images.

The only artifacts of motion would be that portions of the desiredcropped region 2322 may not be available for some scenes. This isillustrated in the figure where one portion 2326 is unavailable, asindicated by the hatching shown. In the played-back video, this sectionwould likely show as a blacked-out regions of the overall frame.

In general, these tilted memory access operations incur significantbandwidth penalties using conventional memory (like DDR). Pulling out acropped rectangular section of memory is common and is usually performedusing a ‘strided’ memory operation, where an equal number of pixels isread from each row. But when pulling out a tilted area from memory, adifferent number of pixels must be read from each region of memory. Forexample the number of pixels to read from row 2324 is smaller than thenumber of pixels required for row 2325. Furthermore, even after thepixels are read, they will preferably be reassembled into a squareorientation matching the displayed version of section 2312, which canincur even more processing as they are transformed back into a desiredorientation. These kinds of row dependent reads and writes cansignificantly degrade memory bandwidth.

An alternative approach to solve this problem is to encode the capturedvideo into memory as a function of the real-time head tracking samplingdata. In this case, the frame buffer is filled as a function of adesired orientation. For example, it might be aligned with the gravityvector during storage.

Employing this kind of controlled video capture and storage willobviously require additional processing and bandwidth. But the advantageof this approach is that the data is ‘straightened’ upon capture andonce written, retrieval and playback of steady video will be much easierand faster. This may mean that the processing technologies required onthe display side will not have to be as sophisticated or powerful toview smoothed and aligned video feeds. This strategy will haveadvantages in many applications, especially in the field of wearablecamera technology.

These motion controlled encoding techniques are equally useful forrecording monoscopic or stereoscopic wide field of view video. Thus,anyone skilled in the art of either of these fields could utilize theencoding concepts presented here.

Various embodiments of the present invention may be characterized by thepotential claims listed in the paragraphs following this paragraph (andbefore the actual claims provided at the end of this application). Thesepotential claims form a part of the written description of thisapplication. Accordingly, subject matter of the following potentialclaims may be presented as actual claims in later proceedings involvingthis application or any application claiming priority based on thisapplication. Inclusion of such potential claims should not be construedto mean that the actual claims do not cover the subject matter of thepotential claims. Thus, a decision to not present these potential claimsin later proceedings should not be construed as a donation of thesubject matter to the public.

Without limitation, potential subject matter that may be claimed(prefaced with the letter “P” so as to avoid confusion with the actualclaims presented below) includes:

P1. A wide field-of-view stereoscopic camera system comprising:

a support structure having M arms, where M is an integer equal to orgreater than 2, each of the arms being spaced around L constructivecircles, each circle defining a plane, where all planes are parallel toa horizontal plane, and each of the arms having one or more mountpoints, each mount point coincident with one of the L circles anddefining a radius of that circle, wherein the radius is half of aparallax distance, and each circle defines one of L constructive, rightcylinders, the axis of the cylinder perpendicular to the plane of thecircle;

N digital cameras, wherein N is at least equal to M, each digital camerahaving a lens with a focal point and an optical axis and producing animage encoded in a digital output and having a horizontal field of viewand a vertical field of view,

the digital cameras mounted to the support structure so that each armsupports at least one digital camera so that (i) the focal point thereofis proximate to a mount point on that arm and (ii) the optical axisthereof is generally tangent to the cylinder corresponding to that mountpoint; and

an image processor, coupled to the digital output of each of the Ndigital cameras, that forms L independent stereo view pairs, wherein,for each stereo view pair, the processor forms a left view from thedigital outputs of a first set of the N cameras and forms a right viewfrom the digital outputs of a second set of the N cameras, wherein thefirst and second sets are disjoint.

P2. A camera system according to claim P1, wherein at least one digitalcamera is mounted so that its horizontal field of view is subject to apartial obscuration by an object associated with an adjacent arm, andwherein the image processor compensates for the obscuration at least inpart by a view available from one of the digital cameras mounted on anadjacent arm.P3. A camera system according to claim P2, wherein the digital camerasubject to the partial obscuration is mounted with an outward angle suchthat the partial obscuration is minimized.P4. A camera system according to claim P1, wherein the left view and theright view both cover the 360 degree horizontal field of view.P5. A camera system according to claim P1, wherein 1 or more of the Ndigital cameras are mounted such that the optical axis is not parallelwith the horizontal plane.P6. A camera system according to claim P5, wherein multiple digitalcameras are mounted proximal to at least one mount point such that theirvertical fields of view partially overlap to capture an extendedelevation field of view.P7. A camera system according to claim P1, wherein the digital camerasmounted proximal to mount points corresponding to one constructivecylinder are further mounted at an elevation with respect to thehorizontal plane such that their fields of view are not obscured by anydigital camera mounted proximal to a mount point that corresponds to adifferent constructive cylinder.P8. A camera system according to claim P1, wherein the image processorfurther operates on the left view and the right view, of one or more ofthe L sets of digital cameras, to estimate the distance of items fromthe camera system using stereo triangulation.P9. A method of providing a wide field-of-view image comprising:

a plurality of digital cameras that are each independently operable anda set of camera orientation sensors that are used to make orientationestimates of the camera platform during capture and wherein the imagedata from the digital cameras is fed to digital processing electronicsthat implements a processing algorithm that creates one or more widefield of view images, derived from the image data streams of two or moredigital cameras and also encodes the orientation estimates into theimage data stream that are subsequently written to a memory device.

P10. The method of claim P9, wherein the orientation estimates are usedto control the encoding of the video data into memory to allow moreefficient export of video for future replay.P11. The method of claim P9, wherein the camera system is a wearabletechnology, qualified by a camera adhered to or otherwise mounted to theclothes, including helmets, gloves or shoes, of a human or animal user.P12. An event viewing camera network comprising:

one or more wide field of view cameras that are operable to combine thevideo feeds of independent digital cameras into a wide field-of-viewimage stream, each wide field-of-view camera mounted at ideal locationsto capture video data of an event of interest, wherein each widefield-of-view camera further incorporates digital processingelectronics, a memory buffer, and a communications interface, where thevideo data is stored to the memory buffer and where the network isaccessed by end-users to periodically select and off-load portions ofthe video stored in the memory buffer for permanent capture and possibleplayback of portions of the event of interest.

P13. The camera network of claim P12, wherein the wide field-of-viewcameras are stereoscopic camera systems.P14. The camera network of claim P13, wherein the wide field-of-viewstereoscopic camera system is composed of multiple supersets of digitalcameras such that each superset is oriented at a different parallaxdistance.P15. The camera network of claim P14, wherein the different supersets ofcameras utilize modified camera operational parameters to view the samescene from alternative points of view.P16. The camera network of claim P12, wherein a central processing nodeis included where the video from all of the wide field of view camerasis collected and that forms the primary node of contact with end-usersfor managing user interfaces, specifying video clips of interest anddelivering video clips electronically.P17. The camera network of claim P12, wherein geolocation tags are addedto the captured video to identify the location at which the video iscaptured.P18. The camera network of claim P12, wherein the end-users communicatewith the camera network through the communications interface and controltheir interactions with the system using device software that runs onthe end-user's personal communication device and allows that user toobtain video clips.P19. The camera network of claim P18, wherein the device software allowsperformance factors to be set by the end-user.P20. The camera network of claim P19, wherein the device softwareincludes the ability to select one or more of the following performanceparameters: which wide field-of-view camera from which to capture video,the viewing frustum of captured video to purchase, the frame rate, thezoom factor, the resolution, the compression quality, the output format,or the period of time corresponding to the scene clip of interest.P21. The camera network of claim P20, wherein preview windows for eachwide field-of-view camera that shows a representation of the images tobe captured from that wide field-of-view camera are included as part ofthe device software and the camera system is operable to communicatedata from its memory buffer to the end-user's personal communicationdevice to fill in data to these preview windows.

Embodiments of the present invention may include software that can bestored on a new transitory computer readable medium in addition to orfor use with the disclosed camera platform. The invention may beembodied in many different forms, including, but in no way limited to,computer program logic for use with a processor (e.g., a microprocessor,microcontroller, digital signal processor, graphical, or general purposecomputer), programmable logic for use with a programmable logic device(e.g., a Field Programmable Gate Array (FPGA) or other PLD), discretecomponents, integrated circuitry (e.g., an Application SpecificIntegrated Circuit (ASIC)), or any other means including any combinationthereof.

Computer program logic implementing all or part of the functionalitypreviously described herein may be embodied in various forms, including,but in no way limited to, a source code form, a computer executableform, and various intermediate forms (e.g., forms generated by anassembler, compiler, networker, or locator.) Source code may include aseries of computer program instructions implemented in any of variousprogramming languages (e.g., an object code, an assembly language, or ahigh-level language such as VHDL, Verilog, Cuda, Fortran, C, C++, JAVA,or HTML) for use with various operating systems or operatingenvironments. The source code may define and use various data structuresand communication messages. The source code may be in a computerexecutable form (e.g., via an interpreter), or the source code may beconverted (e.g., via a translator, assembler, or compiler) into acomputer executable form.

The computer program may be fixed in any form (e.g., source code form,computer executable form, or an intermediate form) either permanently ortransitorily in a tangible storage medium, such as a semiconductormemory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-ProgrammableRAM), a magnetic memory device (e.g., a diskette or fixed disk), anoptical memory device (e.g., a CD-ROM), a PC card (e.g., PCMCIA card),or other memory device. The computer program may be fixed in any form ina signal that is transmittable to a computer using any of variouscommunication technologies, including, but in no way limited to, analogtechnologies, digital technologies, optical technologies, wirelesstechnologies, networking technologies, and internetworking technologies.The computer program may be distributed in any form as a removablestorage medium with accompanying printed or electronic documentation(e.g., shrink wrapped software or a magnetic tape), preloaded with acomputer system (e.g., on system ROM or fixed disk), or distributed froma server or electronic bulletin board over the communication system(e.g., the Internet or World Wide Web.)

Hardware logic (including programmable logic for use with a programmablelogic device) implementing all or part of the functionality previouslydescribed herein may be designed using traditional manual methods, ormay be designed, captured, simulated, or documented electronically usingvarious tools, such as Computer Aided Design (CAD), a hardwaredescription language (e.g., VHDL or AHDL), or a PLD programming language(e.g., PALASM, ABEL, or CUPL.)

While the invention has been particularly shown and described withreference to specific embodiments, it will be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the invention asdefined by the appended clauses. As will be apparent to those skilled inthe art, techniques described above for panoramas may be applied toimages that have been captured as non-panoramic images, and vice versa.

What is claimed is:
 1. A wide field-of-view stereoscopic camera systemcomprising: a support structure having M arms, where M is an integerequal to or greater than 2, each of the arms being spaced around aconstructive circle and having a mount point, coincident with thecircle, defining a radius of the circle, wherein the radius is half of aparallax distance, and the circle defines a horizontal plane and thecircle defines a constructive right cylinder that contains that circleand that has an axis perpendicular to the horizontal plane and whereineach of the arms has a length adjustment to vary the distance betweenthe center of the circle and the mount point; N digital cameras, whereinN is at least equal to M, each digital camera having a lens with a focalpoint and an optical axis and producing an image encoded in a digitaloutput and having a horizontal field of view and a vertical field ofview; the digital cameras mounted to the support structure so that eacharm supports at least one digital camera so that (i) the focal pointthereof is proximate to such arm's mount point and (ii) the optical axisthereof is generally tangent to the cylinder; and an image processor,coupled to the digital output of each of the N digital cameras, to forma stereo view pair, wherein the processor forms a left view from thedigital outputs of a first set of the N cameras and forms a right viewfrom the digital outputs of a second set of the N cameras, wherein thefirst and second sets are disjoint.
 2. A camera system according toclaim 1, wherein at least one digital camera is mounted so that itshorizontal field of view is subject to a partial obscuration by anobject associated with an adjacent arm, and wherein the image processorcompensates for the obscuration at least in part by a view availablefrom one of the digital cameras mounted on an adjacent arm.
 3. A camerasystem according to claim 2, wherein the digital camera subject to thepartial obscuration is mounted with an outward angle such that thepartial obscuration is minimized.
 4. A camera system according to claim1, wherein the left view and the right view both cover the 360 degreehorizontal field of view.
 5. A camera system according to claim 1,wherein 1 or more of the N digital cameras are mounted such that theoptical axis is not parallel with the horizontal plane.
 6. A camerasystem according to claim 5, wherein 2 or more digital cameras aremounted proximal to at least one mount point such that their verticalfields of view partially overlap to capture an extended elevation fieldof view.
 7. A system according to claim 1, wherein each of the arms ispivotally coupled to a mounting pole that passes through the center ofthe circle and is disposed perpendicularly with respect to thehorizontal plane, each of the arms having a link pivotally attached to acircumferential support that is slidably coupled to the mounting pole,so that motion of the circumferential support along the mounting poleadjusts the radius of the circle and therefore the parallax distance. 8.A camera system according to claim 1, wherein each of the arms iscoupled to a sliding arm mechanism that allows the arms to move insubstantially radial direction and that includes a rack and pinion, sothat rotational motion of the pinion around its central axis adjusts theradius of the circle and therefore the parallax distance.
 9. A camerasystem according to claim 1, further comprising: a motorized drivecoupled to at least one of the arms, and a controller coupled to themotorized drive, so as to cause adjustment in length of at least one ofthe arms, wherein the controller is coupled to one of the cameras on theat least one of the arms to receive from the camera at least oneparameter of the camera and to cause an adjustment in length of at leastone of the arms responsive to a change in the at least one parameter.10. A wide field-of-view stereoscopic camera system comprising: asupport structure having L sets of arms, where L is an integer greaterthan 1, and each set has at least 2 arms, each of the arms in aparticular one of the sets being spaced around a distinct one of Lconstructive circles, and having a mount point, coincident with thedistinct one circle, defining a radius of the circle, wherein the radiusis half of a parallax distance, and the distinct one circle defines aplane that is parallel to a horizontal plane and a constructive rightcylinder containing the distinct one circle and having an axisperpendicular to the horizontal plane; N digital cameras, wherein N isat least equal to the number of arms in the camera system, each digitalcamera having a lens with a focal point and an optical axis andproducing an image encoded in a digital output and having a horizontalfield of view and a vertical field of view, the digital cameras mountedto the support structure so that each arm supports at least one digitalcamera so that (i) the focal point thereof is proximate to such arm'smount point and (ii) the optical axis thereof is generally tangent tothe cylinder corresponding to that mount point; and an image processor,coupled to the digital output of each of the N digital cameras, thatforms at least one stereo view pair, wherein, for each stereo view pair,the processor forms a left view from the digital outputs of a first setof the N cameras and forms a right view from the digital outputs of asecond set of the N cameras, wherein the first and second sets aredisjoint.
 11. A camera system according to claim 10, wherein at leastone digital camera is mounted so that its horizontal field of view issubject to a partial obscuration by an object associated with anadjacent arm, and wherein the image processor compensates for theobscuration at least in part by a view available from one of the digitalcameras mounted on an adjacent arm.
 12. A camera system according toclaim 11, wherein the digital camera subject to the partial obscurationis mounted with an outward angle such that the partial obscuration isminimized.
 13. A camera system according to claim 10, wherein the leftview and the right view of at least one set of L cameras both cover the360 degree horizontal field of view.
 14. A camera system according toclaim 10, wherein 1 or more of the N digital cameras are mounted suchthat the optical axis is not parallel with the horizontal plane.
 15. Acamera system according to claim 14, wherein 2 or more digital camerasare mounted proximal to at least one mount point such that theirvertical fields of view partially overlap to capture an extendedelevation field of view.
 16. A camera system according to claim 10,wherein the digital cameras mounted proximal to mount pointscorresponding to one constructive cylinder are further mounted at anelevation with respect to the horizontal plane such that their fields ofview are not obscured by any digital camera mounted proximal to a mountpoint that corresponds to a different constructive cylinder.
 17. Acamera system according to claim 10, wherein each of the arms in atleast one of the L sets of arms has a length adjustment to vary thedistance between the center of the circle and the mount point.
 18. Acamera system according to claim 10, wherein the image processor furtheroperates on the left view and the right view, of one or more of the Lsets of digital cameras, to estimate the distance of items from thecamera system using stereo triangulation.
 19. A camera system accordingto claim 10, wherein the processor, responsive to a signal received bythe processor, provides a selected one of the stereo view pairs as anoutput.
 20. A camera system according to claim 10, wherein each of the Lstereo pair of views is formatted to be compatible with a 3D displaytechnology.